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ABSTRACT 

This monograph describes che activities of the second 
year of a 5-year program of Transition Institute research to 
understand and assist evaluation of demonstration transition programs 
for students moving from special education to work and adult living. 
A case study approach was used to create a description of two 
transition programs, based on the perspectives of the various kinds 
of program participants and sensitive to the situational context in 
which tne program operates. A Santa Barbara (California) School 
District project focused on transition issues of Mexican-American 
parents, problems with interagency agreements, the importance of a 
shared view of transition, and the perceptions of project staff 
concerning the ideal evaluation report. The second case study, of a 
community college program in Grays Harbor, Washington, was selected 
because it involved private sector coor-iinators who were not 
previously experienced in special education, an economically 
depressed community with high unemployment, and an outside evaluator 
whose previous work was known to be of high quality. The case study 
deals with the notion of local success versus national model, the 
effects of overpromising in grant applications, the evaluator 's role 
in project development, generalizability, and reasonable expectations 
for evaluation. (JDD) 
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The following pnnciples guide our research related to the education and employment of youth and 
adults with specialized education training, employment, and adjustment needs 



• Individuals have a basic right to be educated and 
to work in the environment that least restricts their 
right to learn and interact with other students and 
persons who are not handicapped 

• Individuals with varied abilities, social 
backgrounds, aptitudes, and learning styles must 
have equal access and opportunity to engage tn 
education and work, and iife-long learning 

• Educational experiences must be planned, 
delivered, and evaluated based upon the unique 
abilities, social backgrounds, and learning styles of 
the individual. 

• Agencies, organizations, and individuals from a 
broad array of disciplines and professional fields 
must effectively and systematically coordinate their 
efforts to meet individual education and 
employment needs 



• Ind'viduals grow and mature throughout their lives 
requiring varying levels and types of educational 
and employment support 

• The capability of an individual to obtain and hold 
meaningful and productive employment is 
important to the individual's quality of life 

• Parents, advocates, and friends form a vitally 
important social network that is an instrumental 
aspect of education, transition to employment, and 
continuing employment 
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The Transition Initiative 
In 1983, the U.S. Office of Special Education and Rehabilitative 
Services (OSERS) announced a priority in the area of the transition of 
students in special education from school to work and adult living. This 
priority came on the heels of the discovery that despite efforts to 
individualize instruction, to maximize integration, and to teach functional 
skills, special education students were not faring well upon leaving the 
mandated services of public education. High dropout and unemployment rates, 
long waiting lists, and dependence upon families and social systems were 
commonly cited problems associated with this population (Edgar, 1987; 
Hasazi, Gordon, & Roe, 1985; Mithaug & Horiuchi, 1983). 

These findings led to a re-evaluation of the role of special education 
in preparing students for adult life; the lack of linkages between educa- 
tion, adult services, and private industry; and the need for coordination of 
services at state and local levels. Although these concerns are raised in 
P.L, 98-199, the first reauthorization of the Education of the Handicapped 
Act to mention transition, the regulations and guidelines for provision of 
services were not clearly specified. Instead, a considerable amount of 
discretionary funding was authorized to support a number of model 
dem^^nstration programs across the country that were proposing or 
implementing innovati/e practices in th^ area of transition. By 
implementing and evaluating these model demonstration programs, individually 
and collectively, it was hoped that the federal government and the field 
would be able to: 

1. give an accurate picture of the status of transition services in 
the United States; 

2. identify areas of success, failure, and need; 

3. inform policy and funding decisions as they arise in Congress; and 
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4, disseminate information on successful transition projects to other 
states and localities, thus expediting the growth of transition 
services. 

Recognizing that somr? assi5^tance would oe required to upgrade the amount, 
quality, and conduct of evaluation activities, monies were authorized in the 
form of a contract to provide funded projects with technical assistance in 
evaluation design, analysis, and reporting. 

The contract was awarded to the University of Illinois at Urbana- 
Champaign to create the Transition Institute, a five-year program whose 
purpose is threefold: (a) to provide technical assistance in evaluation to 
more than 100 federally funded model demonstration transition projects 
across the United States; (b) to monitor the impact of the Federal 
transition initiative; (c) tc carry out research on effective strateg-^is for 
conducting and evaluating transition programs. 

The Purpose of the Monograph 

This monograph describes tfie activities of the second year of a 
five-year program of Institute research designed to understand and assist 
evaluation in the model demonstration programs. During the first year, 
extensive literature review and a panel of research methodologists, 
evaluation practitioners and theorists, and special educators were used to 
identify and examine the issues associated with evaluation in the transition 
projects. These issues included: local capabilities for evaluation, the 
importance of contextual ity in evaluation reporting, the application of 
evaluation standards, the practicality of responsive vs. preordinate 
evaluation designs, and the utility of different types of evaluation 
information. Papers were written on each of these topics and published in 
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the monograph. Issues in Research on Evaluation in Transition (Stake & 
DeStefano, 1986). 

During the second year an attempt was made to obtain local perceptions 
about the extent to which the issues identified in the first year were 
present at the project level and xo assess their impact on the activities 
and attitudes of the project staff. Two strategies were used to explore 
these issues. First, a paper-and-pencil survey ms sent to all past and 
present project directors to collect information on each of the issues 
identified previously (Stake, DeStefano, & Bergin, 1988). It was agreed 
that although this information would be useful in summarizing the common 
experiences of project directors, additional attention should be given to 
understanding the unique patterns of interaction among and within families, 
schools, and communities that seem to characterize transition programs. A 
case study approach was used to create a description of each program that is 
based on the perspectives of the various Kinds c*" program participants and 
is sensitive to the situational context in which the program operates. The 
methodology was chosen because it allows for the inclusion of multiple 
perspectives in studying complex phenomena. The outcome of the case study 
is not valuation, or causal understanding, but enlightenment, and in some 
cases bewilderment concerning the issues involved in developing, operating, 
evaluating, and replicating a model demonstration project. Two case studies 
are presented in this monograph, preceded by an overview of the model 
demonstration process for readers who are unfamiliar with it. 

The Model Demonstration Process 

The model demonstration process comprises six phases: (1) development; 
(2) implementation; (3) operation; (4) evaluation; (5) dissemination; and 
(6) replication/continuation. The development phase is undertaken for the 
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most part before the proposal is submitted for funding. Proposal^ are 
evaluated in terms of the clarity and completeness of the design, evidence 
that the commitment of responsible parties has been secured, the 
qualifications of key personnel, and the quality of the evaluation plan. In 
practice, many of these criteria are not in place until the implfimentat^on 
phase, which begins as soon as the proposal is funded: project personnel 
are hired, meetings are held, and interagency agreements are ei*tabl ished. 
During the operation phase, program staff are responsible for doing what 
they said they were going to do— carrying out project activities and 
upholding promises that were made in the proposal. Especially in the area 
of transition, this is likely to entail orchestration of activities among 
diverse groups such as employers, public schools, community colleges, adult 
service agencies, and parents. 

Evaluation can occur during or after the operation phase. In 
response to the criteria in the original request for proposal (RFP), 
evaluation designs are necessari''y preordinate and tend to follow a 
goal-based approach. According to federal guidelines, the purpose of the 
evaluation is to document the effectiveness of the project in meeting its 
stated purpose and the purposes of the federal initiative. As stated in an 
RFP taken from a transition competition, the evaluation should document "(1) 
the grantee's progress in achieving the objectives in its approved 
application; (2) the effectiveness of the project in meeting the purposes of 
the program; and (3) the effect of the project on persons being served by 
the project." Although evaluation data may well be used formatively, the 
primary purpose is to provide evidence of program effectiveness for 
accountability purposes and to represent the projects to others interested 
iri replication. 
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Dissemination may occur early in the life of a project, as it makes 
itself known in the milieu in which it operates. Brochures, press releases, 
video segments, and presentations may be developed during the first few 
months of funding to attract a clientele, rally community support, or 
establish a political base. In the later stages of a project, dissemination 
activity is again heightened, this time with the purpose of informing key 
parties to assure project continuation, to disseminate data on project 
effectiveness, and to encourage replication and continuation. 

During repl ication , the viability of the model project is tested in 
another setting. It is an important test of the validity of the 
intervention, because the moc'al demonstration projects are supposed to 
provide models that can be adopted by other localities. Replications can be 
purposive, initiated and monitored by the parent program, or they can be 
adventitious, when other sites adopt part or all of the program without the 
assistance of the parent program. Replication can be complete or partial, 
formal or informal, planned or unplanned. Continuation past federal 
funding should be planned for throughout the life of the project. As 
federal sponsorship decreases and ends, the project must make plans for its 
continuance in part or in entirety through state and local funds. Program 
continuation is based upon documentation of program effectiveness, 
availability of funds, political viability of the program and its staff, 
project visibility, and need for services. 

The "Unofficial" Story 

Evolving simultaneously with the official life cycle of the federal 
model demonstration project are the day-to-day activities in which students 
learn, teachers teach, meetings are held, products produced, and services 
delivered. Project directors must carefully maintain and balance both local 
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and federal perspectives during the lives of their model demonstration 
programs. This balance is not always maintained; at times local needs may 
override or undermine capabilities for fulfilling federal expectations. On 
the other hand, project directors who take seriously their responsibility 
for federal reporting and data collection may feel that they are 
shortchanging service delivery. 

Project directors express the feeling that they are behind schedule 
before they begin. The preordinate program and evaluation designs proposed 
in the grant application fall prey to politics, staffing problems, and 
delays in funding. The possibility of accomplishing all that was specified 
in the original proposal decreases as the realities of the situation are 
dealt with in the implementation phase. On the other hand, most project 
directors say that they have managed to accomplish a great deal in a short 
amount of time, that major project goals have been fulfilled, and that 
existing services have been improved as a result of their programs (Stake, 
DeStefano, & Bergin, 1988). 

The discrepancy between what actually occurs in a transition program 
and what is captured in an evaluation of that program is troublesome to 
program staff, funding agencies, and others trying to understand the complex 
phenomena of transition. Federal expectations of a preordinate, 
outcome-oriented, goal -based evaluation approach make it difficult for 
projects to take credit for unanticipated "getting ready" accomplishments or 
to accommodate setbacks. Disallowing unintended accomplishments and 
setbacks while promoting the evaluation of attainment of stated goals 
produces a situation in which programs that are successful in the eyes of 
staff and consumers appear to have fallen short by the standards of the 
funding agency. 
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The Case Studies 

In this monograph w& present case studies of two transition programs. 
Although the methodology remained the same, the issues that arose during the 
case studies varied. In his case study of a Santa Barbara, California, 
School District project, Terry Denny was concerned with the transition 
issues of Mexican-American parents; the problems with interagency agree- 
ments; the importance of a shared view of transition; and the perceptions of 
project staff concerning the ideal evaluation report. In his study of a 
community college program in Grays Harbor, Washington, Robert Stake found 
himself focusing on the notion of local success vs. national model, the 
effects of overpromising in grant applications, the evaluator's role in 
project development, general izability, and reasonable expectations for 
evaluation. 

There are differences between the projects and the persons who studied 
them 'In terms of community characteristics, time on site, familiarity with 
the topic, and background and role of the evaluator. There are also some 
similarities. Both projects dealt with placing people in employment 
settings, and both were located in schools or community colleges. In 
addition, both projects were attempting to build a program where none 
existed and \\o alter existing structures while doing so. Both projects were 
nearing the end of their funding cycle and facing the possibility that they 
would be re-funded. 

The life of a model demonstration program involves a balance of 
research and practice, innovation and practicality, and generalizability and 
site specific neeob. These projects are funded with the intent that in a 
relatively short period of time they will implement, evaluate, and 
disseminate a model of service delivery that will be viable in part (but 
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preferaDly in its entirety) in other settings. The need to validate the 
accompli rhments and to understand the workings of such programs is great; 
the strategies that will enable us to do so are difficult to identify. In 
the case study that follows, a close look is taken at the role, conduct, and 
use of evaluation in transition programs, with the hope that increased 
understanding may lead to improved policy and practice. 
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Evaluation at Grays Harbor 

Robert E. Stake 
Transition Institute 
University of Illinois at Urbana-Champaign 

The Transition Project at Grays Harbor, Washington, provided a special 
opportunity for studying the role of evaluation in the development of a 
model site for providing transitional services to youth with handicaps. The 
coirenunity was economically distressed; the institutional setting was 
complex. For recent graduates near the bottom of their ciiSS the employment 
picture was bleak. Educational, welfare, and job assistance agencies failed 
to reach many needing help. 

Needed in Grays Harbor County was an initiative. It came in mid-1985 
from an unusual source, a five-year-old for-profit service agency, 
Organizational Architects, Inc. Co-directors Mike Taylor of Westport and 
Carol Rich=»rdson of Olympia sought and obtained funding from the U.S. Office 
of Special Iducation and Rehabilitation Services (OSERS) to develop a model 
transition program. They sought and obtained cooperation from a school - 
district-based pupil services cooperative, the local community college, and 
an existing parents' support group. Together in a two-year periou they 
identified 151 unemployed young adults, brought 35 to the college for 
courses, and saw 11 through to employment. 

Federal funding carried with it an obligation for formal program 
evaluation. A vigorous critical analysis was promised by program 
personnel. A highly respected specialist entered into contract to assist 
the effort. The evaluation circumstances were in many ways similar to those 
at a hundred other transition projects across the country, and similar to 
those of projects expected in the foreseeable future. In each case there is 
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a commitment to evaluate, an effort to review procedures, and some kind of 
assessment of progress and impact. 

And in almost all cases, efforts to evaluate fail to tell us much of 
what we want to know. Among the problems are: differences in federal and 
local expectations, lack of attention to contexts, and uncertainty about how 
flexible evaluation designs should be. This case study was undertaken to 
examine the role of evaluation in the conduct of such projects. 



The Case Study 

My study of evaluation roles and issues depended on a thorough 
understanding of the transition program at Grays Harbor. Throughout the 
1986-87 school year, I examined the workings of the program, the setting, 
the network of individuals and agencies which created special transition 
services in this rural and coastal county. I start the case report by 
introducing personnel: 

Tom Owens, evaluation specialist. Northwest Regional Educational Lab; 
Carol Richardson, co-director. Organizational Architects, Inc., Aberdeen, 
WA; 

Mike Taylor, co-director. Organizational Architects, Inc., Aberdeen, WA; 

John Harp, vocational instructor, 'irays Harbor Transition project; 

Gene Shermer, Academic VP, Grays Harbor CommunHy College; 

Joseph Malik, President, Grays HarOor Community College, Aberdeen, WA; 

Gary Higashi, Vocational Director, Learning Assistance Center, 6HCC; 

hc.ry Carthum, Director, Grays Harbor Pupil Services Cooperative; 

Dave Sander, Grays Harbor Pupil Services Cooperative; 

Claudia Self, Grays Harbor Advocates (parents support group); 

Rosemary Polosaari, Grays Harbor Advocates (parents support group); 

Pat Mahalik, Transition Advocate, Grays Harbor Advocates 

Lee Busco, Youth Employment, Grays Harbor County; 

Ken Colburn, special education teacher, Amanda Park School; 

and other agency people, employers, work supervisors, and trainees--and 
Robert Stake, evaluation researcher. University of Illinois, because in 

gathering my data I too became a minor character in the happenings. 

The Transition Program 

Two people calling themselves Organizational Architects, Inc. designed 
and implemented a l laborative assistance "model" linking nine rural school 
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districts, their pupil services cooperative, the state's Division of 
Developmental Disabilities and Division of Vocational Rehabilitation, and 
community-based organizations, with the community college and "private 
sector." The aim was to identify and meet personal and employment needs of 
area youth with mild handicaps (learning disabilities) who were out of 
school and ineligible for other aid. 

Co-directors Richardson and Taylor sought out and talked to the youth 
referred to them, as many as they could find in the 1869 square-mile Olympic 
peninsula county. For basic skills training and workplace orientation they 
brought 25 to the Grays Harbor Community College campus just south of 
Aberdeen. As in most special education programs, males outnumbered females, 
in this group 19 to 6. The trainees ranged in age from 17 to 30. Ten had 
finished high school. One by one they left the Project, averaging a little 
less than four months of training and entry-level employment under super- 
vision. When the Project officially terminated during the summer of 1987, 
six of these 25 were known to be not working, another six had dropped out of 
sight, and the remainder were employed in such work as lumbering, building 
maintenance, engine repair, institutional housekeeping, and clerking. 

The Project brought vocational educator John Harp to campus to teach 
job preparation skills. He taught about choosing a job, applying for work, 
interviewing, and about relationships between employers and employees. He 
often elicited long discussions of problems such as observing petty theft by 
fellow employees. The students were enthused to be "college students." 
t\arp gave less attention to a student's learning difficulties or weakness of 
social skills than most special educationists would have. 

Disagreement as to instructional strategy and the fact that more than a 
year passad before the first students were brought to campus contributed to 
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rather lukewarm participation by the Pupil Services Cooperative and thus, by 
the local schools. In contrast, though slow to emerge, an enthusiastic 
parent group calling themselves the Grays Harbor Advocates provided hundreds 
of hours of community contact with students and parents. Response to the 
transition services from Community College administrators was mixed, some 
noting reluctance of their supporters to welcome students with learning 
disabilities to campus. Still, they provided one of the instructional 
strategies and, ultimately, an institutional home for courses. 

The Grays Harbor Transition Project was funded by the federal govern- 
ment for a two-year period in the amount of $140,000 to create a model 
transition service. Announcements in the Federal Register had encouraged 
proposals from private sector, for profit, entrepreneurs. One question to 
be evaluated was whether or not such a company as Organizational Architects, 
Inc. could improve upon ordinary arrangements in the difficult job of 
coordinating public institutions and volunteers trying to help school 
leavers with handicaps. 
The Setting 

Through the eyes of participating trainees the program consisted of two 
or three people and a couple of classrooms in the vocational wing at Grays 
Harbor Community College. A mile south of the bridge leaving the Aberdeen 
business district, further still from the wood-loading docks at Hoquiam, the 
college nestles in an evergreen hillside. Students here are the Grays 
Harbor "Chokers," a choker being one who locks the chain around a load of 
timber. A Bunyonic lumberjack marks the entry. Not much else about the 
College beckons youngsters having marginal experience in high school. 
Faculty members and administrators are proud of the College's academic 
program. 
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Most folks in this vast mountain, coastline, rain-forest county live 
around the pie-shaped harbor (see map, following page), though few are close 
enough to see the gray whales frequenting its mouth. At the Westport 
lookout station, tourists count whale watching and great expanses of beach 
among the pleasures. The mountains hide the villages, Quinault, Grisdale, 
Artie. Montesano has the court house and a delicatessen much to my liking. 
A second pie-shaped wedge, the Quinault Indian Reservation, presses inland 
from Queets and Taholah. Punning north and south, US 101 cuts the county 
quietly; US 12 runs east and west, more like slash and burn. 

Aberdeen and Hoquiam are union towns. Still, more of the county's 
trees are cut and hauled by independent rigs than by Weyerhaueser. For 
would-be workers, knowing someone somewhere is important. The project staff 
seemed to know the people and the tv^rriiory. In the eyes of a trainee, the 
transition staff people came by car or called by phone. They knew the boss 
or supervisor, often spent a while chatting about recurrent mists, 10'^ 
unemployment, or the grocery-worker strike. 

It's a rural place. The sea is near, but people live close to the 
land. The work ethic is strong. Many young people leave home reluctantly. 
Even to take a job in the next town is a major decision, especially for a 
daughter. The welfare roll is long. Whether or not a youngster's job will 
cost more in welfare deduction than it brings in is an issue parents debate. 
Program Activities and Alliances 

What I describe here happened within the second year of the project, 
ending July 1987. Young adults no longer in school and showing signs of 
mild intellectual handicap and needing employment were identified. From 
high school records and through the grapevine the list grew, ultimately 
reaching 151. The College, the parents' group, and the Pupil Services 
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Cooperative, all made referrals. Advocates Claudia Self and Rosemary 
Polosaari, themselves parents of special education children, tracked down 
many of the youth and brought their records up to date. 

According to the evaluator's report, contact was made and initial 
information obtained on 103. Educational skills were tested and work 
experience discussed. About a i:hird were referred to agencies such as 
Timberline Opportunities, a sheltered workshop. Eligibility requirements 
were not strict. Most had been identified in school as special education 
students, but not all. One, for example, had had a brain injury after 
leaving school. If talk with the young adult confirmed an interest in 
further preparation and suitability for entry-level vocational work, in they 
came. 

Their interests and self-esteem were explored. One girl wanted to sell 
women's wear, but only the fashionable. A young man already knew small 
engine repair, but hesitated to expose himself to an employer's expecta- 
tions. Some expressed reluctance to leave home. Some had unrealistic ideas 
about careers and educational opportunities. 

With assistance from the Project, all enrolled in basic skills (reading 
and math) courses offered by the College. Additionally, in groups of eight 
or ten they learned about characteristics of employment generally and about 
specific employment conditions there in the county. John Harp's instruction 
was group-organized, casual, based on vocational materials he had developed 
earlier. His approach was personal istic, contrasting with the behavior- 
analytic "cooperative education" approach vigorously advocated by Gary 
Higashi, vocational coordinator at the College. Little instructional 
collaboration Letween Harp and Higashi occurred, but perhaps little was 
needed. 
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After a slow beginning, collaboration at administrative levels did 

occur. The College, the Advocates, and the Cooperative participated in 

policy, tactical, and information-sharing meetings. Assistance of the State 

Division of Vocational Rehabilitation (DVR) and Labor and Industry (L&I) 

office was obtained. Even at the end, full-blown cooperation from any of 

the participating agencies could not be taken for granted. For them, of 

course, transition goals lacked priority and were not always pursued as 

they would have pursued them. But after extensive effort, individuals 

within the agencies were helping out. In the evaluation report John Hart 

was quoted as saying. 

In the past DVR wouldn't work with the Community College. 
State agencies satisfy their needs in their own formal way, i.e., 
paperwork. Now they are doing very well, in fact, tremendous. 

The Pacific Mountain Private Industry Council provided JTPA (Federal) funds 

which enabled new employees to work for a month at no cost to the employer. 

This was a key matter in getting actual work experience for the students. 

And it cccasionally resulted in a regular job. Each employer agreed to 

identify a person to supervise each trainee. Time and work evaluation 

sheets such as the one on the following page were kept. This sheet (an 

elaboration of a form suggested by the project evaluator) is for a student 

who did maintenance work at a retail outlet during the first cwo weeks of 

January. He accrued 16.5 hours and was paid $55.27. A couy of his bank 

deposit slip also was kept in project files. He continued lo work at least 

four additional weeks. Students could remain with the program as long as 

they liked. They could request additional work try-outs. Several remained 

in the program for more than 7 months. Most stayed about three. 

Project co-directors Richardson and Taylor spent their time, half-time 

each officially, facilitating the collaboration. Carol handled more of the 
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central office function, Mike the face-to-face cantacts with employers, 
supervisors, and trainees at work. During the final 10 months of the 
project, evaluator Tom Owens identified almost 500 institutional contacts. 

A special education transition program on campus runs into strong 
commitments to lecture-hall type instruction. John Hart said, "Many on 
campus were uninformed about learning disabilities." A key event occurred 
in January of th^'t second year. Chatting about the upcoming Superbowl , Hart 
and his students made a lunch bet. Hart lost, and ingeniously invited 
College President Joseph Malik, DVR and L&I administrators, and several 
other agency people to join them. Lunch lasted several hours. The 
administrators appeared to move toward advocacy; the students appeared to 
regard themselves with greater respect. It seemed at the time a turning 
point. But a second event may have nullified the first. In late spring a 
trainee was accused of stealing at the worksite. Enthusiasm lost something 
of its edge. 

By late spring (1987) tha Project was nearing its end. As intended, 
plans were debated for transferring coordinating responsibility to one of 
the existing agencies. College autnorities seemed reluctant to continue 
services to this "special population." As he described the College to me. 
Vice President Gene Shermer indicated that vocational courses had only 
recently become 20% of the offerings. As to students with handicaps he 
said, "We a''^e novices in this." seemed favorably disposed to the work of 
the Transition Project, and by May seemed ready to assume responsibility for 
its instructional activity. But it did not happen. Discontinuation was not 
discussed with Richardson, Taylor, or Harp. John Harp was hired by the 
College as Developmental instructor, but no further transition courses were 
offered. According to Harp, on his own he reworked his c tivity packet for 
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enhancing job-readiness skills and these were made available to students for 
self-study by Pat Majalik of the Advocates group. JTPA funding for 
entry-level employment support continued to be available through Lee Busco 
of Youth Employment--as it was during the two years the Project operated. 

Earlier, with a few months to go, wanting to continue and needing funds 
but cognizant or OSERS policy against posi -project continuation funding, the 
co-directors submitted an application to 'start" another OSERS project 
locally. (Originally the co-directors had wanted funding for three years 
but got it for two.) The new proposal was not funded. Formal evaluation of 
the initial project had not been completed, so an evaluation report did not 
figure in the competitive review. 

This transition project had been funded to be a model project. I was 
not unusual for Mike Taylor to speak about the importance of keeping records 
of issues and problem-solving efforts so that educators elsewhere (the 
allusion usually to a rural site, once he said, "down in Tuscaloosa, 
Oklahoma") could be guided by the dif-'^^'cul ties and successes of this 
project. Mike savored the idea of writing up the project's "life story." 
With Richardson and with us visiting evaluation specialists, he discussed 
issues and efforts--but as of this writing, 4 months a^ter project 
termination, the logbook, the diary, the essays^ had not appeared. A final 
report was "on the way," but it seemed unlikely that it would greatly 
facilitate generalization. The key thing is that seldom did the staff 
examine situations in a way directly leading to an understand ng of what was 
particular to this Gr^ys Harbor venture and what might generalize to many 
settings. Right or wrong, the staff felt that the Grays Harbor Transition 
Project would be most valuable as a model if it succeeded, and everyone 
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dwelt far more on making it a success for a local constituency than on 
making it a learning model for a national constituency. 
Project Evaluation 

In the original Organizational Architects, Inc. proposal, Richardson 
and Taylor included an extensive plan to evaluate the process and product of 
transition service in Grays Harbor County. I photo-reduced that two-page 
section of the proposal and include it here. Whether or not their fulsome 
promise to validate their model was serious, realistic, and optimally suited 
to the Project is a key issue of this evaluation case study. 

Among institutional resources identified at the end of the proposal is 

the Northwest Regional Educational Laboratory of Portland, Oregon. The 

authors stated that the Lab's "expertise will be utilized to develop 

appropriate evaluation instruments and an on-going evaluation mechanism 

which will allow for changes during program implementation." Working on 

employabil ity of disadvantaged youth, the two co-proposers previously had 

had a productive relationship with Tom Owens of the Lab and looked forward 

to his help again. (Harry Carthum had taken steps to bring in a University 

of Washington evaluation specialist, but Owens was the Richardson-Taylor 

choice.) Owens clearly had the expertise cited above, a predilection for 

good instrumentation plus cautiousness about overly-fixed designs. A 

flexible design allows attention ti 3w issues and understandings as they 

appear. In March, Owens spoke to me of his frame of reference. 

I've been doing [federally funded] case 'itudies with problems 
pertinent to those of youth in Grays Harbor County, especially 
reflecting the economically troubled fishing and timber business 
there. Vocational education, special education, basic skills--the 
evaluation studies overlap. You get maximum mileage for the 
evaluation dollar by doubling up in this way. 
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Although the Project's federal contract was signed in October of 1985, 

its evaluation subcontract was not signed until August of 1986, Owens 

provided reports on the following 1986-87 schedule: 

Sep 30 Evaluation Progress Report 

Oct 15 Evaluation Instruments Package 

Nov 13 Evaluation Instruments Package (Revised) 

Feb 20 Evaluation Progress Report 

Aug 30 End-of-Year Evaluation Report 

Richardson and Taylor wanted Owens as thei/ evaluator partly because 
they knew he would help them think through their operations. They knew he 
would not be impersonal and excessively objective in serving as outside 
assessor of project quality. They wanted an evaluator who measured 
accomplishment of both staff and students, but not one who attended 
exclusively to quantitative data. They expected technical assistance from 
him. Owens had impresses* them as knowledgeable and informative about 
research on vocational education for special populations. They looked to 
him as management consultant as much as program evaluator. 

Although its federally funded Transition Institute provided technical 
assistance on evaluation and other matters, OSERS had expressed no objection 
to having evaluation specialists provide technical assistance as well as 
summative evaluations to individual projects. Taylor invited Owens to the 
Annual Meeting (of transition project directors) in Washington, D.C. to 
assure that the Grays Harbor project evaluation would be "in compliance" 
federally as well as useful locally. 

Owens said, "They were quite open, letting me choose the design and 

instrumentation most appropriate." He recognized that Carol and Mike needed 

more evaluation than OSERS required. He told me, 

The^ planned to get JTPA funding too so I was interested in 
documenting student impact as well as possible. I could have 
"gotten by" with a more streamlined approach. Federal 
requirements were relatively minimal. At the Annual Meeting they 
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had a whole bunch of questions to be covered in the final report. 
I had already drafted the evaluation plan but I went back and 
revised it to make sure the instrumentation was comprehensive 
enough. 

Owens called for the following management-informatic, records: 

1. Learning contract (for individual trainee at worksite), 

2. Timesheet (for individual trainee at worksite), 

3. Counseling notes (for observations cf work arrangement), 

4. Employer evaluation (of the Transition Project), 

5. Work performance evaluation (for individual trainee at worksite), 

6. Applicant information sheet (for individual trainee to fill in), 

7. Student data card (for staff to keep), 

8. Individual training pla.i, 

9. Program completers survey, 

10. Log of contacts with agencies and individuals— 
and, deleting his earlier recommendation of the ETS Youth Career 
Development Test , he identified the following evaluation instruments and 
data sources: 

1. Referral correspondence, 

2. High School and Beyond and other research literature, 

3. Brigance Diagnostic Inventory of Essential Skills results, 

4. CTB/McGraw Hill Test of Adult Basic Education , 

5. Interviews by the external evaluator, 

6. Trainee writing assignments, 

7. Washington Occupational Interest Survey , 

8. General Aptitude Test Battery . 

The 11-month evaluation contract was budgeted at $5,000. During the 
period, Owens visited the site four times. Relationships between parties 
remained cordial throughout the period. Owens depended on the Project staff 
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to keep records and administer the tests needed. His final report was based 

considerably on his earlier progress reports. Its preparation was delayed 

by late returns of trainee activity. 

In his first evaluation progress report Owens described his September 

site visit. Citing Dave Sander of the Pupil Services Cooperative, he 

described the Project's special population: 

... The 9 high schools in the area graduate about 30-50 resource 
students a year, of whom about half don't fit into employment. 
They are not low enough to be considered developmental ly disabled 

but not high enough to qualify for some othe*" programs Most 

of the youth who will be in this program ar.. specific learning 
disabled— way below grade level in math and reading, some social 
behavioral problems, a few mildly retarded, none are physically 
handicapped alone, low socio-economic status, some drug problems, 
and disfunctional families. These youth usually had one to four 
periods a day in resource classes, several periods in a regular 
class, and some went to work-study. Small businesses are usually 
unwilling to take on these youngsters [even] with JTPA help. The 
youth need a person with whom they can identify . 

Owens also wrote of the Community College situation, where no-cost adult 

basic education was offered to all with a high school diploma, which many of 

these trainees had. A number of the referrals to ^!.e Transition Project 

were from the College itself. The College had no staff members traineH in 

special education. 

For his part, Owens indicated an optimism in working at the site. With 

eyes sparkling, he cited staff criteria for success of the Project, 

including: "change in student self-concept, independence (ability to handle 

oneself), and the number who enter employment or further training." In his 

February progress report Owens summarized major evaluation-related events 

and noted that the instrument package data were being collected by John Harp 

on all students. During his February site visit, Owens chose a student 

situation to inquire into and wrote the following: 

The student interviewed began the program in November, 1986. 
His father asked the community college about classes for the son 
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who has a reading disability. The student is interested in 
automotive repair and wanted to learn to do job interviews 
better. Although he had learned job interviewing techniques while 
in a high school career class, that occurred in his freshman year 
and was in a large class then. Through the Transition Project he 
is learning it better, in a smaller class setting, and the 
interviews with employers are videotaped and evaluated later. 

He had previous mechanic experience working for his father in 
a family-owned motorcycle business but is now working 25 hours a 
week through the project in car and diesel mechanics. He has a 
chance to do everything from tune-ups to major overhaul because 
the shop is small with only 3 employees. He will receive JTPA 
money for 250 hours of training and hopes to get permanent 
employment there after that. Although most students in the 
program take a bus, this student has his own car which he uses to 
get to work and school. He likes the small size of the program 
and is making personal friends ... with whom he socializes on 
weekends. He is also learning from watching others in the program 
and he now helps other students to use the computer to prepare a 
resume. The things he likes best about the program are the chance 
to get a job and the instructor's help in developing his job 
interviewing skills. A concern he has about the program is that 
some of his friends in it, he feels, are ready for job placement 
but the instructor is overloaded with work and hasn't time to 
place some of the students yet. 

Other aspects of the project also appeared vibrant and on-target in Owens* 

February progress report. 

I will describe Owens' final report later in the section titled 

"Evaluation as Technical Assistance." In the remainder of this section I 

wish to point out and comment on evaluative judgments in Owens' final 

report. It included evaluative comments from John Harp, some employers, and 

a w students, but almost nothing summarizing perceptions of quality as 

seen by Tom Owens himself, the evaluator. The following were the exceptions: 

Establishment of close working relationships among the 
various agencies serving special needs youth was a major objective 
of the p.oject and one where the co-directors at Organizational 
Architects, Inc., did an excellent iob. I attended several of the 
interagency monthly meetings and found them to be well organized 
and useful. The monthly agency contact logs kept by the project 
staff indicate particularly frequent contact with the Grays Harbor 
Pupil Service Coop, the Advocates, and with cooperating employ- 
ers. Also, monthly contacts were made with the Division of 
Vocational Rehabilitation, Labor and Industries, the local high 
schools, and parents of students. Early referrals of youth while 
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still in high school was one area that did not get adequately 
developed in the project. 

Collaboration [with the private sector] was excellent. The 
project had training slots it couldn't fill. This is especially 
surprising in a community with high unemployment. 

[As to making] the project to be fully transitional, it needs 
to identify and select handicapped students needing employment 
assistance while they are still in high school. 

The scarcity of evaluative statements was attributable, I think, to the fact 

that Owens got fewer data on the late spring students than he wanted, and 

saw interest in the Project waning. Perhaps he no longer felt fully 

inclined to close out the study with summary statements of effectiveness and 

impact. 

In interviews with me he was rather expansive and in general favorably 

impressed. Here are some of those judgmental observations: 

... The types of jobs they are getting for the people seem 
really good to me. These are not "dead end" jobs. When you deal 
with special needs populations it's a challenge to get them jobs 
having real potential. 

... The proposal called for them to be working very closely 
with school districts, to have contact with student even at 9th 
and 10th grade levels so there would be continuity in transition 
into this program. That seems to have fallen by the wayside. It 
IS not getting much attention at all. 

... If the Community College would pick up John Harp's part 
of the program there is od possibility you could divide the rest 
of the program among the institutions now participating. They 
could keep the thing going. There could be life after federal 
funding. If anyone could do it well it would be Mike and Carol. 
They have the vision. They can pull together things like that. 

... I don't see them interested in general izabil ity. If they 
get a new project funded, which is equivalent to the continuation 
of this one, then they would be interested in it. 

Owens was sensitive to a number of the major evaluative issues identified in 

the next section, but chose not to report on them formally. For one reason, 

his data pool was shallow. And issue development was not required by his 

formal contract, nor paid for. he did not orient as much to the vitality 
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and genera^ izabil ity of the enterprise as an outsider would expect. His 
orientation was local, his information was gathered to assist Taylor and 
Richardson. In the long and short of it, his wcs a project-based study, not 
a federal -program-based study— though he was careful to equip the Project to 
satisfy its federal requirements. 
Research on Evaluation 

During the fall of 1986, with help from my Transition Institute 
colleague, Lizanne DeStefano, I selected the Grays Harbor Transition Project 
as a case study for examining several key evaluation issues. We reviewed 
140 proposals and selected three sites, the others at Danville, IL and Santa 
Barbara, CA. The Illir.ois and California sites were chosen partly for 
researcher ease-of-access. Projects to be studied were selected that 

1. had immediate contact with students; 

2. utilized a variety of community work placements; and 

3. involved schools, state agencies, community, and private sectors; 
but as a group gave us 

4. geographical spread; 

5. experience with a variety of handicap conditions; and 

6. a variety of contextual issues. 

In selecting each of the cases we kept in mind a number of issues our team 
identified last year as needing study. (See Issues in Research on 
Evaluation in Transition which I edited for the University of Illinois 
Transition Institute). These issues are identified in the next section. 
The Grays Harbor Project was attractive because it brought in (a) private 
sector coordinators who were not previously experienced in special 
education; (b) an economically depressed community with high unemployment; 
and (c) an outside evaluator whose previous work was known to be of high 
quality. 
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At the November 1986 Annual Meeting I approached Taylor and Owens and 
found them willing to consider participation in the case study. Knowing 
that progress in the Grays Harbor Project had been disappointingly slow, 
Mike Taylor was reluctant to increase its complexity, but he foresaw that my 
case study might facilitate documenting the project for distant readers. 
The arrangement was confirmed later after I spelled out my needs and Taylor 
and Owens had had a chance to discuss it with Carol Richardson. I pointed 
out that I did not intend to evaluate the Project and that I would be more 
attentive to its evaluation activity than to its transition activity. 

I visited the Grays Harbor area for several days in March. I observed 
John Harp's class and an interagency meeting, met several of the students, 
visited several worksites and talked with supervisors, and interviewed 
administrators at the Cooperative, the Community College, and a rural 
school. I discussed the effort and the issues at length with Richardson and 
Taylor and separately with Owens. In August, after Project completion, I 
returned, intending to learn of reactions to the final report, but found it 
only in draft form and not yet seen by interested parties in the county. I 
interviewed Richardson, Taylor, and Owens about project termination and 
College plans to continue the transition services 

Not a special education researcher, I v/as unfamiliar with some of the 

program issues. I identified what I considered to be key programmatic 

problems, but concentrated on what I considered to be key evaluation 

problems. I frequently relied on Owens, the project co-directors, and 

DeStefano to correct my interpretations of the transition effort. Some 

misperceptions probably remain in this account. Even though the primary 

purpose here was to increase understanding of the role of program evaluation 

with transition projects, a correct and thorough understanding of the 

program was needed. The program issues I conceptualized are presented in 

the sections to follow, after a discussion of the evaluation issues. 

06 
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The Issues 

Many who study quality in education find it useful to distinguish 
between progt am issues and evaluation issues. Issues are major ideas about 
which people disagree. People will disagree about how the program is 
operating and how it should be evaluated. The co-directors of the Grays 
Harbor Transition Program and their outside evaluator identified both kinds 
of issues. In my case study here I was trying to examine evaluation issues, 
so I will take them up first. 
Evaluation Issues 

What questions will best organize our thinking and lead to greater 
understanding of the role of evaluation at the Grays Harbor site? Four 
major clusters of questions became prominent. After presenting these issue 
questions I will discuss each cluster with detailed reference to evaluating 
the transition work at this site. 

l. Overpronising . What is a reasonable time period for building a 
base in the community and developing such a model? Unexpectedly slow 
progress in Grays Harbor early on was voiced as one reason for less than 
full interagency cooperation later. Should disappointing accomplishment the 
first year be chalked up to faulty operation or to excusable overoptimism? 
Is this an instance of the general tendency of those who request funding to 
make proposal claims and plans which overpromise what can be accomplished? 
Do the directors, sporting high ambition, direct work away from modest 
accomplishment toward the unattainable? 

2. Criteria of Success . Should job placement be the primary 
criterion for program success? Is duration of employment just as 
important? At Grays Harbor we saw concentration on basic skills training, 
though the proposal emphasized employment criteria. On what standards 
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should the evaluator focus? Do absolute standards (here, assurance of 
eighth grade skill levels) or relative standards (a gain for each trainee) 
serve quality control equally well? 

3, Technical Assistance ^ Did the external evaluator effectively 
balance evaluation as technical assistance and evaluation as guarantor of 
accountability? Did the assistance role nullify the obligation to identify 
shortcomings? Did the evaluator increase project dependence on outside help 
or assist the staff in becoming self-reliant? What is the proper role of 
evaluation reports in application for contribution or new federal funding of 
transition projects? 

4. General izabil ity . Did the Project co-directors use their 
opportunity well to create a model and leave a record potentially useful to 
other speciaJ education transition service personnel? The extensive 
evaluation commitments set forth in the Project proposal were far from fully 
covered in the evaluator' s final report. The evaluator complied with the 
provisions of his contract. As of November 1, 1987, the Project staff had 
not submitted a report indicating--among many things--the general izabil ity 
of their approach, but continued to prepare one. Will that report serve the 
desire of OSERS to identify general izable models? 

Other evaluators (or meta-evaluators) would of course have identified a 
different array of issues, at least in part. After reviewing our Transition 
InstUute work of the first year, we selected this site and I traveled to 
Aberdeen expecting to investigate certain issues. Seven had been identified 
by our Institute Task Group and (as mentioned earlier) had been presented in 
our 1986 report. Issues in Research on Evaluation in Transition . The 
seven: 
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1. Discrepancies between federal expectations and local capabilities, 

2. Special characteristics of "transition" that affect evaluation, 

3. Inattention of evalu£tors to broad societal values, 

4. Inattention to professional standards set forth by evaluators, 

5. Inattention of evaluators to epistemic elements in evaluation, 

6. Flexibility of evaluation designs, and 

7. Regard of situational context in designing the evaluation. 

Given our work the year before, the first three Grays Harbor evaluation 
issues - overprom^sing, criteria of success, and technical assistance - were 
perhaps predicted on discrepancies between the values and standards of 
various stakeholders, particularly federal sponsors and local agencies. The 
fourth issue, generalizabil ity, included and extended our attention last 
year to situational context in the collection and analysis of evaluation 
data. More now on the four evaluation issues I studied at Grays Harbor. 

Overpromisinq . As indicated earlier and as reproduced above, the 
proposal drafted by Richardson and Taylor to obtain OSERS funding promised 
extensive program evaluation and validation. Their transition services 
model was to be thoroughly examined and validated. And indeed, work was 
frequently and conscientiously reviewed by the co-directors and their 
collaborators, and by the outside evaluator as well, ity own inquiry 
increased evaluative attentions by the staff and local sponsors. More than 
an ordinary self-scrutiny occurred. The proportionate share of total work 
spent in formal and informal evaluation was large. 

And yet very little quality assurance f,r val.dation was documented. 
The promised "life story" of the Project has not, at least to date, been 
written. It would not stretch the truth to say that the impact of the 
project remained undocumented. If the comparison of alternative organiza- 
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tional strategies occurred (as promised in the proposal), the results were 
not made available to potential model-users. In the closing months major 
participants in Grays Harbor disagreed about the quality of the collabora- 
tive effort. Comments remained largely off the record, some too painful for 
public expression. Evidence that the model "worked" was not documented. It 
was not simply a failure to comply; the proposers promised much more evalua- 
tive assurance than their resources could have delivered. 

The pr:)posers promised to work with nine districts and 30 students. 
They did. The major goal of che program was to 

^^v elop a model that will provide effective planning and 
progr am, development to meet the service and employment need s of 
hariuicapped individuals as they leave school. 

They did develop such a model. They did correctly see their task as one of 

model building. They recognized that the model needed validation, but did 

not assign or reserve resources for carrying out a validation study. 

According to Owens, in writing the proposal (and therefore in promising to 

validate the modeO Taylor and Richardson were conceptualizing a three-year 

project. After receiving funding for two years, they could have indicated a 

reduction in the validation expectation--but did not. When I interviewed 

them with less than half a year to go, they appeared to feel that a smoothly 

running project transferred to ongoing institutional control would 

constitute a validation. 

Almost a year i4r.d a half passed before the first students were being 

trained. Outside evaluator Tom Owens did not identify this as a failing, 

D/t Pupil Services Cooper?:ive Director Harry Carthum did. Were the 

services tardy? Individual observers did not agree. Proposal timelines 

were not precise (and should not have been), but much emphasis within the 

proposal was given to the time-consuming business of validation. Proposal 
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readers and evaluators could reasonably expect student enrollment during 
Year 1. Taylor was convinced that the pace of the Project was outside his 
control--they were pushing ahead in as many ways as they knew how. He felt 
they deserved credit for overcoming enormous inertia. But overcoming 
inertia was not a federally recognized criterion of success. And the 
proposers had not indicated that building a base of agency cooperation would 
be difficult. 

Criteria of Success . Richardson a:id Taylor based some of their 
thinking about validation on the fact that even for model-building projects, 
federal orientation to project success was directly keyed to job placement. 
They spoke of having their model win endorsement of the National Diffusion 
Network. They did not doubt that almost", everything they did related to 
preparing youngsters for work, helping then; get acquainted with work, and 
increasing likelihood they would remain at work. But the co-directors 
doubted that the quality of what they did in identifying needy youth, 
arranging agency cooperation, and helping youngsters deal with a panoply of 
obstacles would be highly correlated with percentage of youth placed. They 
had more than one objective to pursue. 

We all know that manv factors determine job placement. Taylor and 
Richardson knew they could be distracted from their service and educational 
obligations if they concentrated on placement. They recognized multiple 
criteria by which prjject success should be based— and resented the 
oversimplification of having any single indicator represent success. Job 
placement was seen to be not as important as federal rhetoric indicated. 
Overcoming inertia, for example, was also important. And reasonable people 
in different responsibilities disagree about how important each is. 
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Federal support was provided and received under a contract to produce a 
model service arrangement for handicapped youth. To be successful in 
federal eyes, to satisfy the contract, the Grays Harbor Transition Project 
needed to establish a smooth and durable collaboration among agencies, to 
place into employment a substantial number of youngsters, and to demonstrate 
that the arrangement would work e^Sirw; -.e. Most of the time the Project 
staff was concentrating on more immediate goals. 

For evaluation theorists the general question is one of honoring 
different, even sometime, contradictory, criteria and standards held by 
different constituencies. Here, the federal sponsor and the local operators 
had different views of how murh should be accomplished. The proposal was a 
poor guide, it lacf'ed specificity and credulity. As is the practice. 
Neither local nor feleral people treat proposal statements as contractual. 
Hyperbole is expected. The document is intended to win funding, not to 
present realities, not to state a final set of expectations. 

Most designers of evaluation models, working along such lines as the 
Joint Standards for Evaluating Programs, Materials and Instruction , 
propose ways of negotiating initial expectations. A few evaluators employ 
what Malcolm Parlett and David Hamilton called "progressive focussing" or 
emergent designs. Even with sophisticated designs, negotiated and emergent, 
the problems of conflicting criteria and standards do not go away. It could 
be that the present situation, one of taking specification within proposals 
and evaluation obligations only half seriously, is optimal. What is clearly 
needed is a better process for revising expectations along the way. 

Evaluation as Te ch nical Assistance . After a year of little progress, 
seeing little help from the Uni,ersity of Illinois Transition Institute, the 
Project co-directors chose to emp.oy an external evaluation specialist. 
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They selected Tom Owens, They recognized they did not have the expertise 
themselves to carry out their validation commitments. They had promised 
validation because the RFP called for evaluation and because they believed a 
model should be validated. But the main reason they selected Owens was 
because they were confident he could help them carry out their immediate 
management responsibilities. Whether or not they objected to the evaluator 
Harry Carthum wanted did not become clear. They had worked with Owens 
previously and found him knowledgeable, ingenious, and easy to work with. 
From him they expected technica- assistance. 

In Owens's words, "Evaluation and technical assistance go hand in 
hand," He offered both. He recognized lack of expertise within 
Organizational Architects, Inc, but a willingness to learn. Far less than 
what was needed could be done. It was only a $5,000 subcontract, so it was 
not highly important to the Northwest Lab, but it fit nicely into the Lab's 
and Owens's own service mission, Owens wanted to help. He wanted the 
Project to succeed, and he wanted to write a credible and useful evaluation 
report. He calculated that for $5,000 he could commit perhaps 20 days of 
work, most at his office in Portland but perhaps four or five in Aberdeen, 
He could spend little time gathering data himself. He would rely mostly on 
the Project staff to obtain what he needed, H^ foresaw the paperwork; an 
evaluation plan, a progress report or two--each consuming perhaps a couple 
of days; and a final report, perhaps taking four. He would keep in touch by 
phone, analyze the data provided, and give Carol and Mike whatever advice he 
could. 

Even though the co-directors knew they eventually needed "summative" 
data on outcomes, Owens's orientation and theirs was for "formative" 
evaluation. They wanted to be guided along the way by his observations. He 
provided a 63-page "instruments package" in mid-October, then revised it a 
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month later. For his 14 evaluation questions Owens indicated the record 
forms and tested that could be used. Richardson and Taylor adopted many of 
the f rms, but urged him to devote attention to development of the model, 
not just to measuring student readiness for employment. They pared down the 
testing to "pre and post" administration of the Test of Adult Basic Skills . 

Owens's final report was a 17-page single-spaced document. Designed 
around 12 of the 14 instrument-package questions, it provided a descriptive 
and interpretive overview of the project. He described interagency activity 
and participant characteristics, noting that participating students referred 
by the College had consistently lower basic skills gains than students 
referred by other agencies. The answers to the 12 questions were based 
largely on staff interviews made several months before project termination. 
The report was submitted several weeks after project termination. Taylor 
and Richardson gave it close attention, but others apparently did not see it. 

What Owens provided through his evaluation subcontract was reasonable 
in three ways: the kind of help he provided, the amount of work he did, and 
his direct and indirect reflections on the merits of project 
accomplishment. He offered formative evaluation more than summative 
evaluation, helping the co-directors examine the issues they faced. Even 
though he produced little information and rather limited interpretation of 
data provided him, Owens devoted as much time and as much professional 
scrutiny as $5,000 buys from the information services industry. He did not 
really provide an assessment of impact and worth of 1 » transition services 
in Grays Harbor, nor of the quality of the model, but he made it possible 
for a careful reader to see difficulties faced and overcome and the extent 
to which local youth with handicaps were assisted in gaining employment. He 
provided pretty much what the staff needed and could pay for. 



Model Programs 38 



Owens fulfilled his contract. Together with Richardson and Taylor he 
defined his role more as technical consultant than accountability auditor to 
policy research. He served his immediate employers well. How well he 
served the Grays Harbor community and the American society is not clear. He 
did not study issues from the point of view of the collaborating agencies 
and institutions. Informally he shared his concern about issues most 
pertinent to American education and the workplace, issues raised in broad 
sweep by OSERS evaluation requirements. But he did not evaluate the Project 
in those terms. Whether he should have is not clear. Technical assistance 
may be the proper emphasis for such an evaluation subcontract. More study 
of v/hat each of an evaluator's options accomplishes is needed. 

General izability of Model Programs . Tom Owens did not provide what 
the original proposal called for in the way of validation of the transition 
model. No evaluator could have, whatever the subcontract size. For $50,000 
or $100,000 Owens could have studied the validation issues raised and 
gathered pertinent data, but the degree to which this model would be 
appropriate for adoption across the country still would not be known. That 
becomes known only when a model is tried out in a variety of circumstances 
under close watch. The reality even then is that local staff members will 
immediately adapt the model, influencing and extending it so that the 
original model survives only in part. 

Long after the project terminated Mike Taylor talked about writing its 
story in a way that would be useful to others interested in the model. 
November arrived, the report was ^aid to be nearly written. A good final 
report would be useful. But even if Taylor were to tell the story 
effectively, it would not, and could not, be a validation of the model for 
use elsewhere. 
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The Organizational Architects, Inc. people did not keep their 
proposal's promises to validate the model • A serious matter? Perhaps, The 
proposal becomes the legal contract and a basis for evaluating success, but 
does not reflect subsequent revision of purpose, schedule, and funding. It 
was unclear from this proposal whether validation was part of a two-year 
plan or three. Governments and other sponsors seem to have little way of 
dealing with unrealistic proposals. Here the promise was to provide 
understanding--and little (to date) has been provided. Apparently no one in 
Washington, D.C. was distraught because the Grays Harbor transition model 
and what had been learned disappeared in the mist. 

Who is to blame? The work of the federal bureaucrat is to translate 
legislation into regulation and opportunity, to keep programs running, not 
to learn from them. Others are to learn, but who are they? We have clearing 
houses and research laboratories, repositories of learning. The records may 
get filed, but seldom retrieved. Neither theorists nor practitioners are 
avid gleaners of vicarious experience. Few believe the models created by 
federal services-delivery projects are worth examining. Some are, but 
almost no one can learn a small project's important complexities. The 
system for identifying which cases should be paid attention to is extremely 
weak. Professional expectation is low. If prob ems or effects can be 
aggregated, then the tally may be entered in some matrix. If a story can be 
promulgated simply, such as through the National Diffusion Network, then it 
may be. But the applied research system is not working. Still, many Mike 
Taylors are optimistic. 

As things stand* the burden of learning belongs to the next generation 
of project directors. These persons too have little time to search for 
models. But more important, in these times they have little expectation of 
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needing them or ledrning from them. The models thac appeal to them are 
comoinations of own experience and ingenuity. New directors occasionally 
will study what others have done, but somehow they know that for anyone to 
understand the nuances of a model in a way that will circumvent obstacle and 
increase success requires a worthy model in the first place and an expert 
record keeper. 

Individual models will generalize some, but not as much as the name 
"model" suggests. How extensive the generalizability probably varies with 
many characteristics of professional staff, community, and student body. 
Often the most to be expected from an outstanding model is suggestion. How 
the evaluator should examine a model to determine its suggestive properties, 
its possible uses to diverse users, is not well understood. 

In midspring, 1987, chatting in a conference room in Olympia, I asked 

Tom Owens a question, one I considered the $64,000 question: "Does Grays 

Harbor really have a model program?" He said: 

Well, it's a unique effort. It is business people getting a 
school district, a community college, and an advocacy group 
working together on transition. And it's doing it well now, 
working against barriers common in many communities. Using JTPA 
funding they have placed 13 students on jobs in the community 
and they hope to get another bunch working in April. But the 
real push is to keep the training [at the community college] and 
support [from a parents' advocacy group] working, working hand 
in hand, after OSERS funding runs out. 

Transition Project Issues 

The [valuator's Questions . The conceptual ground underlying Tom 

Owens 's evaluation study was established by the questions he presented in 

the Instruments Package dated November 13, 1986. As I indicated earlier, 

the co-directors had called for an extensive evaluation and validation 

perspective. Owens outlined a modest package of observations and 
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measurements. The facets of the Project which Owens felt important for 
immediate evaluation are illustrated by the following 14 (paraphrased) 
questions: 

1. WJiat major past problems have kept youth with handicaps from 
successful transition to employment and further education? 

2. What are the trainee characteristics of the young people in this 
project? 

3. How does the Project staff identify unique needs in each participant 
and develop an individualized action plan ? 

4. How did participants hear about the Project and what were their 
reasons for joining ? 

5. What program strategies are used to increase participants' basic 
skills and employment and training options? 

6. Do students make a significant gain in basic skills ? 

7. Do students achieve adequate employabil ity skills ? 

8. Are training and employment related to study interests and abilities? 

9. Do students eventually make a positive transition ? 

10. How effective is the interagency collaboration in meeting the needs 
of participants? 

11. What attitudes toward the project are held by participants, parents, 
employers, and staff? 

12. How effective was the private sector involvement in the training and 
employment of Individuals with handicaps? 

13. In what ways can Project improvement occur? 

14. Was the Project's Transition Model developed suitably for adoption and 
adaptation? 
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Owens raised the basic questions expected by OSERS. He spoke of his own 
strong interest in issue questions #6, #10, and #12. As with most 
evaluators with psychology or special education backgrounds he was 
particularly interested in characteristics of the ^^tudents. During the year 
he became attentive to several additional issues: 

15. What is the quality of jobs they get? 

16. From what referral sources do the good trainees come? 

17. Which trainees gain most in basic skills? 

18. Hjw much communication is there among the agencies? 

19. What are the pros and cons of a college as the training site? 

20. What responsibility does the high school have during the year or so 
after the special education student leaves? 

Though all 20 of Owens's questions are important, they did not direct 
attention to what turned out to be the shortfall in transition processes at 
Grays Harbor. The Project encountered serious difficulty with interagency 
cooperation, with trainee misbehavior at work site, and with transfer of 
transition responsibilities to the agencies. Informally, the evaluator 
considered these matters but did not give focus to them nor speak of them in 
the formal evaluation report. Though chosen partly because of his 
flexibility, Tom Owens stuck to his original issues. 

Program Issues Deserving Research . When the Grays Harbor Transition 
Project is looked at from policy-research perspective, a set of issues 
somewhat different from Owens's set emerges. Entering with a frame of 
reference shaped by the Illinois Transition institute* s seven evaluation- 
theory questions (described earlier in "Evaluation Issues"), I found the 
following program issues to be pertinent i.i the Grays Harbor Project. 
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1. Were relationships among participating agencies good? Was co-director 
confidence in these relationships misplaced? Was the Project less 
effective because the Pupil Services Cooperative prevented access to 
youngsters still in high school? Was there resentment against the 
large funding granted to a recently created for-profit agency? Did 
this agency effectively vanish, leaving a well-functioning program to 
be run by existing institutions? 

2. Is the Project a redundancy, repeating services already provided by 
existing agencies? Is job preparation the key need of the targeted 
students? How much emphasis should be put on increasing students' 
ability to get their own jobs? To what extent is general vocational 
preparation useful to all? Do these young peocle need more of what can 
be offered by special education specialists? 

3. Were the right students served? Were native Americans and certain 
others greatly in need of transition services those least likely to get 
help? In working primarily with referrals, was the Project helping 
only those easiest to help? 

4. Did the co-directors work effectively at building a general izable 
model? Was program general izability diminished because employment 
placements relied heavily on a co-director's social network? 

5. In lowering its entrance requirements and downscaling instructional 
offerings to serve these needy youngsters did the Community College in 
any way jeopardize the quality of its established offerings? Did the 
alliance help make* the College less attractive to local youngsters than 
it had been and thwart community efforts to keep them at home? 

6. Observed were a strike of grocery store workers, changes in state 
definitions of welfare, and other economic developments locally. Were 
these contextual factors needing programmatic attention by the Project? 

^2 
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As a researcher studying the evaluation work at Grays Harbor, I made a 
few tries to relate those six program issues to evaluation. Relationships 
among the participating agencies were typical, much like what we find in 
other communities. Any agency anywhere is busy, with little resource for 
doing more; is somewhat suspicious when a new program nudges into its 
territory; and is resentful of implications that existing services are not 
what they should be. Interagency facilitation deserved to be a major 
criterion of success. Here, special educators admired neither the Project's 
administrative prowess nor its training orientation and did not allow 
Project interaction with students still in high school--yet special educator 
Dave Sander worked diligently to make the Project a success. The visitor 
here could still hear words of disbelief that Organizational Architects, 
Inc. was funded rather than existing agencies, more because of its lack of 
track record than because it represented the private sector. 

In many eyes, the Project was a redundancy. Yet that was not an 
evaluation issue. Many local youth with handicaps were not being helped by 
existing agencies, and with a new nexus, additional youth were served. But 
many people supposed that had the money been available to existing agencies, 
the same or better services would have occurred. When the Project closed, 
the community college bowed out; also, an existing JTPA contractor continued 
to provide entry employment assistance. Many of the students had previously 
had years of basic skills remedial training and some job preparation. It 
remained unclear that a new institutional configuration was needed. Still, 
transition needs were real. Young pi^ople*s attitudes toward training and 
employment do change, and a number of those served here appeared to make 
good use of the opportunity. With little arrangement for person-specific 
traininn before the trainee reached the worksite, the educational 
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contribution here appeared modest, yet Project attention to youth needs 
appeared not redundant. All this deserved evaluator review. 

The Project co-directors recognized their commitment to model 
building. To them that meant setting up the best interagency collaboration, 
providing the best training, and arranging good work opportuni ties--and then 
letting others know how the obstacles have been overcome. They did not 
approach the problem of lea^^ning what the limits of av^iceptance would be, 
under what conditions would something work or not work. Their choice was 
probably wise, for they had even less competence to validate their model 
than competence to create it. They built a workable model, of questionable 
durability, of unknown generalizability. As their evaluator pointed out, 
what they did well was to get several agencies to provide good transition 
services to youth who needed it. Should evaluation expect more? 

Of the foregoing issues, how many should Tom Owens have addressed? 
It's a matter to be negotiated, but ultimately it is his choice to make, 
jeyond the routine how-much and how-often and how-good questions, the 
evaluation specialist faces a large choice of issue questions. Many issues 
must be ignored. Never is the evaluation complete, [valuators have neither 
the time nor the talent to go beyond a modest inquiry. Some evaluators do 
not address any important issues. Owens did not violate the norm. He 
fulfilled his contract, working harder at helping the staff understand their 
situation than attesting to their accountability. Neither Richardson and 
Taylor's way nor Owens's way seems the way toward breakthroughs In 
transition services, but it was a way to serve needy young adults in a 
needy community. 

What are reasonable expectations of evaluation? Not broad 
* gener'il izatlon about whav works! As Lee Cronbach, Egon Guba, and Carol 

Weiss have concluded, DeStefano and I also have concluded that an aggregate 
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of local project evaluations will neither determine the success of the 
national effort nor the reasons why what happened happened. Evaluation 
budgets are too small to support the best inquiry we could do, but even our 
best efforts seldom answer the most important questions. From project to 
project, dissimilarities among aims, tactics, and settings defy efforts to 
generalize. Other than contributing to the census of what has occurred when 
and y,here, federal questions are seldom a'^swered by project-based 
evaluatioii. Such was the case in Grays Harbor. 

Evaluation plays another role. Evaluation requirements tell 
professionals about the expectations of government agencies, and thus 
indirectly, the expectations of the people. For example, by emphasizing job 
placement criteria, an economic frame of reference is established. By 
requiring standardized data, the staff's self-serving anec^Iotes become less 
tenable. Obviously there are more direct ways of stating national goals and 
preferred styles, but the obligation to evaluate is one of the ordinary ways. 

But the real service of formal evaluation requirements is in enhancing 
inquiry undertaken by local stakeholders, particularly the project staff. 
It appears that the nore con^,cientious educators, those already bus> 
reviewing project quality, are the ones who can be helped most. Whether 
long experienced in special education, as was Harry Carthum here, or little, 
as were, at the outset, Carol Richardson and Mike Taylor, directors find 
that good evaluation requirements prod and guide self-study. Yet require- 
ments can be misdirecting and enervating. Specifying objectives, for 
example, is useful to a point, but a drain or. work if always on the agenda 
and a straight jacket if overly heeded. 

Whether stated in application procedures, provided through technical 
assistance, or generated by outside consultants, evaluation requirements can 
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be powerful facilitators of good project management. Too little meta- 
evaluation research has been done to demonstrate which kinds of requirements 
strengthen projects and which kinds weaken. In this project, evaluation 
requirements did not appear to contribute positively to the first year's 
organizational efforts. These did not facilitate interagency planning nor 
resolve differences in perspective of transition services needed. Could a 
different set of OSERS evaluation requirements done so? 

When outside evaluator Tom Owens was brought in, he targeted formative 
aims more than summative. He (a) helped e'-.tablish a management information 
system, and (b) moved toward the co-directors' dissemination aims (which 
ultimately were not pursued). His aims exceeded OSERS requirements, but 
fell short of "model" validation, Owens 's interpretation of requirements 
facilitated getting ordinary work done and keeping records most likely to be 
needed. Informally he encouraged inquiry into fundamental transition 
issues, many of which Taylor and Richardson already were discussing. Owens 
could have done more, but what he did fit the size and content of his 
subcontract Concentration on federal needs may someday hance pursuit of 
fundamental generalizations. But we know not how. 

Evaluation played both formal and informal roles in the Grays Harbor 
Transition Project. The formal requirements set up by the original RFP 
lackel specification but allowed essential local adaptations. The formal 
requirements set forth through technical assistance from the Transition 
Institute helped establish traditional management routines, orienting more 
to compliance than elucidation. Commitnients to validation set forth in the 
Project proposal posed unrealistic obligations, but these were largely 
ignored by everyone. The outside evaluator set forth formal evaluation 
requirements which further operational ized management routines and 
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informal evaluation requirements whicli supported thoughtful review of what 
was happening. All in all, a nice oalance between asking too much and too 
little. Still, in its #1 aim, model building, the Project fell short of 
expectation. Better evaluation requirements possibly could have made a 
difference. Probably not, but possibly. 
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Evaluation Notes from the Santa Barbara, California 
Transition Services Project 
Terry Denny 
Transition Institute 
University of Illinois at Urbana-Champaign 
Fieldwork 

I lived in the Santa Barbara area for about 12 weeks during the fall 
of 1986. The data from which this story was derived were gathered during 
a five-week period within my stay. More often than not I would have a 
single interview on a given day and usually have about five per week. In 
this unintentionally leisurely manner, about six to seven "full" days were 
devoted to field interviewing. 

It can be frustrating work when after a week of waiting one meets a 
highly recommended informant only to find that what you came for is only a 
shell— a husk of meaning. But fortune smiles too, when a boilerplate 
interviewee proves full of life. A few would use my presence to chop off 
more heads than Alice*s queen, and others would bask me in the saccharine 
shades of their spreading apathy. But I truly loved them all: they are 
all a part of this story. We share common goals: to achieve immortality; 
to make it home each night. 

From a list of over 40 potential interviewees created by the director 
and assistant director a core group of 16 were selected. Two refused. 
Seven additional area people not on the original list were added as a 
result of i^ecommendations or leads provided by the interviewees. 
Scheduling of interviews proved difficult. (One person was rescheduled 
four times.) Often it took two weeks to get an appointment. Travel time 
to the interviewee was a major cost. 
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In sum, 30 people were interviewed: three project -^riministrators , 
five teachers, three parents, two students, six agency representatives, 
two business persons, two university professors, two project consultants, 
and five state department special education specialists. Some of the 
interviewees provided multiple perspectives of the project; for example, 
one person was a parent of a severely handicapped 19-year-old, a parent 
trainer, and a professional educator. 

Most of the interviews lasted about 45 minutes. A few ran an hour 
and a half. None was tcpe recorded. Everyone was given the opportunity 
to add to or retract their comments by phone after the interview. (Two 
people added considerably to original observations.) One obvious short- 
coming of this field report is its coverage of business/industry 
participation in the project. I shall comment on this later. 

My fieldwork coincided with crucial events in the life of the pro- 
ject: for example, the last two months of the project's funding. No 
re-funding was in sight. And tragically, the spouse of the project 
director passed on after a lengthy illness. I was given free reign 
throughout my stay. I began by reading two project documents and learned 
that after the first six months of the two-year project, four action 
components were settled on as strategic efforts to be made. 

The Four Strategic Tasks of the Santa Barbara Project 
1. A Transition Cooperative was to be organized to assist the project 
in meeting its objectives. The Transition Cooperative would be 
designed to allow the key (28) agency stakeholders opportunities to 
facilitate transition of special education students into the adult 
community. Collaboratively its members would work to overcome the 
problems identified above and serve as an advisory body to the project- 
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2. An Employabil ity Curriculum Committee was to be formed within the 
school system to prepare special education students for postsecondary 
educational opportunities or employment or both. 

3. An Individual Transition Plan was to be written for each special 
education student prior to leaving school, for example, graduation from 
high school. This plan would be formulated with the assistance of 
appropriate adult service providers in the community' and would include 
assessment information useful to those agencies and the individual and his 
family, recommendations for continued instruction, information about 
specific skill areas still needing development, and resources available in 
the community to meet the goals collaboratively identified by the 
individual, the family, the school system, and probable future providers 
of service. Support services that were not presently available in the 
community and that were critical to the successful transition of the 
individual were to be developed. 

4. A Transition Support Group , composed primarily of parents, v^as to 
be established to provide support and training to parents of youngsters 
either preparing for or in the midst of accomplishing the transition from 
school to adult life. A group of volunteers would be recruited and 
trained to assist individuals with special needs in successfully adapting 
to work situations by providing coaching and counseling individuals on 
ini,.ial job placements. 

The successes of this project for the most part built on good efforts 
and good people known to project leaders. "We went with the winners. We 
only had two years. We had some luck, but the major successes were not 
luck, nor a surprise. We knew where the good people were. This project 
permitted us to get them together." 
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Before turning to the three principal parts of the story of the Santa 
Barbara Transition Project (parents, transition, and evaluation), a note 
on the state investment in special education may aid the reader unfamiliar 
with California's educational scene. 

Special Education in California 

Special education is big business in California. At the bureaucratic 
top is the State Board of Education's 17-member Advisory Committee on 
Special Education. Annually it schedules nine two-day meetings throughout 
the state. An Associate Superintendent directs a state program serving 
378,738 California children with special needs, a majority of whom 
(212,097) had a specific learning disability. Some 5,554 students were 
multihandicapped, 3,280 deaf, 26,805 mentally retarded, 7,289 orthopedi- 
cally impaired, and 160 deaf blind. Special classes and centers serve 
118,432 and resource specialist programs are provided for another 
146,319. A small number attend state special schools (1,013). 

California public school personnel who provide services to special 
education students number 18,142. About 50 percent teach (9,561) in 
special classes; another ?0 percent (5,255) are resource specialists. 

Entitlement funds for instructional programs further confirm the 
big-business image of California special education: $835 million from the 
state general fund are augmented by $271 million from the sta^e school 
fund, $166 million from local school general funds, and $82 million from 
federal flow through. County property taxes contribute another $84 
million. Still other revenue sources contribute further to a whopping 
$1.6 billion annual budget for special education in California.^ 



"Special Education in Ca fornia." California State Department of 
Education. Bill Honig, Superintendent. Sacramento 1986. 
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The $1.6 billion is about 12 percent of the total budget for public 
education (K-12) in the state. About 10 percent of the total 
instructional personnel and 9 percent of the students are in special 
education. In some way or another about every tenth educator, student, 
and dollar is a part of the California special education scene. In 
California, after one is 22 years old one is no longer entitled to school 
services. Heretofore this has meant the SH (severely handicapped) young 
adult was placed principally in sheltered workshops or remained at home. 
More recently, options have included community integrated employment 
situations with the possibility of group home living. 

State commitments to serving kids with special needs are paralleled 
in the Santa Barbara School District. It provides a full range of 
services. Much of that effort will not be reflected in this report. 

Evaluative Remarks about the Project 
This is not an evaluation of Santa Barbara's Transition Project. 
Nor is it a description of that project, nor a study of the project's 
leadership, efforts, and achievements. It 21 the residue of what 30 
interviewees caused me to think about when they talked about their view of 
the Santa Barbara project or their view of transition projects. From time 
to time, I would interject an evaluative question, probe for a value, try 
out an idea provided by another interviewee. 

The interviewees presumably had their minds on what they cared to 
tell me about transition. I tried to keep my mind on what their words 
implied for evaltiating transition. It was an interesting tug of war. 
Sometimes we dropped the rope and thought together about a shared notion. 
More often we engaged in parallel play. 



Model Programs 54 



I am grateful for the carte blanche given me by the Santa Barbara 
project leadership and for the generosity of my informants- They gave me 
time, ideas, stories and--when the gods smiled--an education- 

There may be ideas worth developing that I failed to recognize on 
those pads of yellow lined pages consumed during interviews. And, the 
ones that follow herein may be inadequately developed. C'est la vie. I 
believe the apparent shortcomings of my site visit and subsequent 
interpretation are outweighed by the evaluation issues that the Santa 
Barbara project helped me consider. In the final analysis, the Santa 
Barbara project did not need an evaluation from me as much as I needed 
evaluative ideas from the oanta Barbara project. 

When one hears criticism of project purpose, development, or outcomes 
based on personal dislike of a project participant, it is pretty thin 
stuff. So it was with most of the critical comments about the Santa 
Barbara project. For the most part, the project was seen as having 
delivered on its promises. The meetings were held; the curriculum was 
developed; the ITP's were being written (one prototype was on the 
computer); the present training prototype was created; a process for 
improving interagency agreements has been initiated. Other project 
outcomes could be noted. School district participants were uniformly 
enthusiastic about their experiences. They were enthused about project 
related services, activities, and products, for example, "This project has 
been the best thing that's happened to me as a teacher in five years in 
the district." 

Participants from other agencies, and peripheral informants (people 
who had not participated— but had knowledge about the project) were less 
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enthusiastic, less generous in their evaluation of the project's process, 
products, and future. Most concerns were with what the project did not 
do, with the small number of student placements, with the slow progress of 
business/ industry involvement, and with the lack of genuine participation 
of a particular key agency representative. Note the concern with the sins 
of omission not commission. Many felt the project was too short. "It 
takes five years to start a project like this. Two years is only 
fool ing around." 

In my exit interview with project directors, I was told of hopes of 
having enlarged participant understanding of facilitating change, of 
pinpointing success variables, and of developing strategies for change. 
These were not my foci. I talked with people about how they viewed 
evaluation, how they evaluated, how evaluation had affected their lives in 
this project and elsewhere, and about what the project had meant to them. 
Three large themes emerged: (1) parents, (2) transition, and (3) 
evaluation. 

If what follows is useful to readers interested in educational change 
strategies, it will be a pleasant surprise to the writer, who knows little 
of such matters. If what follows does not raise concern with those 
interested in educational evaluation, I shall be distressed. The shock 
waves I 1elt in Santa Barbara are still with me three months after my 
departure. I shall continue with a comment on context. 
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CONTEXT 

"Every morning I wake up to start another ho-hum day in paradise." 
(Receptionist, Tri-county Regional Center) 

The Santa Barbara project first seemed to be indistinguishable from 
the many school district projects I have reviewed over the years. Lovely 
little town, Santa Barbara, with 78,000 or so residents to enjoy its many 
advantages. The climate is kind to its street people, who number in the 
hundreds, as well as to its annual $320 million agribusiness industry. 

Fou,- privately maintained museums augment the University of 
California campus's contribution to the arts. The city has its symphony 
orchestra and over 20 music organizations which either present 
performances or sponsor music events. 

The transition "coop" is also an elegant operation. It has its able, 
take-charge politically smooth administrator, its comnetent, 
get-things-done second-line administrator, plus a cast of sophisticated 
characters who contributed to my story. 

Throughout my interviews, project innovations were brought forward 
with surety to enable me to see their uniqueness as well as their 
functional merit. Tales were told about the key contributions of this 
person or that. The party line (about the nature of this two-year effort) 
started to emerge after the second interview. "Kids drop off the cliff 
when they leave schooling." And, "Once into adult day programs, they 
never get out." And, "We have only begun— but there is no turning back 
now." And, "Parent involvement is the key. They change the whole nature 
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of an educational project when they are true partners," And, "The 
involvement of corporate stakeholder is the central and most unusual 
dimension of our transition efforts/' And, "It's the way we do our work 
from the bottom up, not the top down that makes this one different-" 
These were themes spoken often, believed deeply, and left unchallenged. 

If an outside evaluator were to scrutinize only the context of the 
aforementioned themes (short-term and long-term goals distinction, parent 
and industry involvement, and process management), a fascinating but 
Incomplete story of the success of this project could be detailed. 

For example, it is unlikely that a native midwestern professor (or a 
D,C. bureaucrat) could easily understand a key dimension of Santa Barbara 
context, one that no one mentioned per se during interviews about the 
project. It had something to do with 987 sunshiny days from August 26 to 
December 9, 1986. It has something to do with the pacific Pacific ocean. 
My first hint came when an interviewee told me to "loosen my tie, get 
tinted glasses and schedule my interviews at Moby Dick's." I loosened my 
tie. My second hint came when a banker talked about her career. We'll 
get to career after a few words about the Pacific. 

Santa Barbara's history is filled with incidents that link with the 
sea. The first European to set foot on the Californian Coast was a 
Portugese navigator, who apparently did so on December 4, 1542, the feast 
day of Saint Barbara. Earthquakes and sailors have left their mark on her 
ever since Richard Henry Dana wrote about Santa Barbara in his novel Two 
Years Before the Mast . So later did Charles Norduff Mutiny on the 
Bounty . 

The pleasant Spanish village adopted its city charter in 1900. An 
ideal climate, with average daily temperatures of 72°F in tne summer and 
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66°F in the winter, and a spectacular shoreline made it a northern 
countryside retreat for the artistic, the religious, the military, the 
ancient, and the rich. Earthquakes regularly reduced their theaters, 
adobe missions, presidios and palatial haciendas to rubble. Immediately 
after tremblers desist, the residents set about to restore and improve 
their monuments. Nothing apparently dismays Santa Barbarbans for long. 
The intrepid sea and quaking earth may contribute to the laid back culture 
one senses soon after immigration. 

This laid back ambience is difficult to track down, but I feel it 
central to project context. To fail to recognize it is to miss an 
integral component of "ho hum" days in paradisiacal Santa Barbara. A UCSB 
university teacher of narrative writing reminded me that John Updike wrote 
about Santa Barbara women rolling down the aisles oi" the A & P, clad in 
swimwear proclaiming their place in the sun. And a state rehab employee 
told me of seeing a well-to-do young man wearing jeans and a rock-and-roll 
T-shirt at a funeral. A bank officer came close to the heart of the 
matter: "I would have been a branch manager [of a large coastal bank] 
years ago if I had chosen to move to L.A.," says Sally, an officer in the 
SB branch. "But, Tm doing just fine. I love my work. I prefer SB rays 
to LA ulcers." She and others let me know that people fight to stay in 
paradise. Sometimes you have to be aggressive to enjoy the fruits of 
being laid back. For example, "It's hard to be a young special education 
teacher in SB when a PhD wants the job, too." 

Two businessmen in their early 30s talked to me about the "no 
problem, mellow attitude of the laid back r.ulture." They observed, "After 
a while you get to see the rush-rush for what it is. There is no sense 
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rushing to tomorrow. Be cool, it will come. And there's no sense 
worrying about debts, problems, pijmotion, mortgages. . .anything can be 
fixed if ic's worth fixing. Life can be fun. It's all pretty simple 
really." 

A graduate student at UCSB mused, "I've been spending 24 years 
learning how to do it [be laid back]. It means being tolerant wir/- ^ 
dash of complacency. It is definitely a UCSB descriptor. It can be 
self-satisfaction. It's ptetty much 'white.' We're here: we've got our 
place in the sun. Being laid back is real positive for enjoying what life 
has to give you and real negative in its arrogance and 
self-centeredness.*' Yes, Santa Barbara is laid back. 

Another ingredient in this transition project that may set it apart 
from other school -district-based transition sites is the size of the 
community. A parent trainer, also the mother of a severely handicapped 
child, moved from Los Angeles to Santa Barbara to "get out of the rat 
race." 

We live where we work. This means we \ »ve neighbors who are key 
actors in companies here. We gu to ch' rch and civic clubs with 
others involved in the larger scenario of making transition a 
reality. And, finally, our size makes it all seem possible, 
manageable. When I lived in L.A., it was extremely difficult to 
see where a project issue began or ended in time, space, 
personnel. The budget was the only factor that seemed knowable> 
definable. 

I shall now turn to a long "footnote" on how educational evaluators 
might get from fieldwork a deeper sense of the local ambience. 

Check the Election Results 

With most of my informants I explored the issue of how an outsider 
could get a sense of the local scene. Their comments and suggestions 
provided the grist for this footnote. A district administrator observed 
that the Chamber of Commerce (C of C) was always willing and able to 



ERIC 




Model Progrdms 60 



provide a sketch of a community, outlining its business, educational, 
recreational, and social resources. I found the C of C literature con- 
firmed aspects of the picture I had after a day or two in the community, 
but it added little and introduced no surprises. The C of C source may 
help ready an outside evaluator prior to visiting the site, 

A California transplant (of 15 years) suggested that evaluators might 
/ read election results and analyses in the newspaper the day after local, 
state, and federal elections. She recalled how it helped her better 
urJerstand the "particuldr political values of the Gold Coast C=il ifornian, " 

So I took her advice and read th? Santa Barbara New Press coverage 
of the November 4 congressional, gubernatorial, an^ local races. Her 
advice proved sound. The analyses informed me about local context in ways 
I found surprising;, I shall mention three. First the local voter turnout 
was almost twice the state average (74% to 42%) and was well above 
national figures (34%), Second, on a proposition to make English the 
official language of the state (svpported by 75% of the voters), a 
majority of the Hispanic ,'Ote went for the proposition. Third, a 
landslide statewide defeat of the Chief Justice of tl^e Supreme Court and 
two of ner co-justices (75% voting for their opponents) after 8 years in 
office was tempered somewhat locally (60% voting for their opponents). 
Community values among registered voters exercising their franchise 
appeared to include: (a) active community participation; (b) chauvinistic 
patriotism; and (c) support for capital punishment. 

These value-iaaen issues were but tips of a complex iceberg. Results 
of county and township supervisor races and of other ballot propositions 
relating to pollution control and salaries of public officials were 
provocative and revealing. In sum, it would appear worthwhile for outside 
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evaludtors to check the local newspapers on the first Wednesday in 
November of even years for hints at local values. 

So much for context and notions about method. The first of three 
major themes (parents, transition, and evaluation) follows. It concerns 
parents o'^ severely handicapped children and minority parents with 
disabled children. 

Parents 

Two of my most enthusiastic, forceful interviewees were mothers of 
severely handicapped children. They valued the project's contributions to 
their own goal of a better life for their children and for themselves. 
They were strong, no-nonsense women with practical goals couched in large 
social realities: "We are used to things folding My son was 
' neurological ly handicapped,' 'developmental ly disabled' .you name 
it... before the project, during the project, and will be after the 
project. The buck stops right here (index finger to breastbone)." 

Many years before one is thinking about the transition needs of a 
high school student there have been grave social issues considered, partly 
caught up in in identification of very young children with special needs. 
Lori and Bill Granger have written a best-seller expose of the school's 
misdiagnosis of their son Alec as being mentally retarded in The Magic 
Feather (Sutton Publ., 1986). Other social agencies fare no better •'n 
their parentdi view of special education in general and its mal treau.fient 
of their son in particular. Parents' attitudes can make or break the 
schools' special education programs. 

At the center of a successful transition project will be the parents 
of severely handicapped students. The more severe the child's disability, 
the more the parent will be concerned with transition. There is a 
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corollary; the older the child gets, the more realistic the parents' 
expectations become. A junior high teacher observed, "It's here they 
start getting scared. They (children) aren't growing fast enough- . .they 
[parents] can't hide it [child's disability] any further." And, within 
tl^ose ranks will be a core group o knowledgeable, purposeful, seasoned 
women. Their educational, religious and socioeconomic backgrounds may 
range widely but ti^eir purpose and character often are very much alike. 

Very few fathers are truly co-partners in the rearing of a severely 
handicapped child. It has been claimed that the divorce rate is far above 
average in these families. The child's needs and demands are several, and 
persistent. In a sense, they define the lives of these mothers.^ They 
deeply impress one with their firm resolution to s*?cure societal support 
for their ultimate goal, the maximum reasonable independent life for their 
children. Their message is clear, simple, forceful: "Everyone deserves a 
choice. Everyone . Now! " 

These women will do what it takes: attend meetings, volunteer time, 
agitate, demand, beg. "But the parents we could count on 20 years 
ago., .really the mothers. . .are not in the cards. Women are out of the 
home. The volunteer pool has shrunk to a puddle." Parents are seen as 
"our greatest resource"; "the driving force behind the project"; "a pain 
in the ass"; "the reality check on our efforts"; and "the slumbering giant 
that will come of age." Jhv^ last view has particular relevance for 
ethnic parents of Sh youth. <ican-Amerfcan parents in particular. 



1. Peculiarly, in Santa Barbara the child is much more likely to be a 
son than a daughter. One counselor (14 years) said he "aKays had 
about 2/3 boys to 1/3 girls in the program." 
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"They Just Don't Come" 

It might be a straightforward task to describe failure to involve 
minority parents in transition project activities if the cause were 
leadership insensitivity or failure to try. Such was not the failings in 
Santa Barbara. Nor ' ^s there an emerging uneasiness such as that 
described by T. S. Eliot in "Little Gidding": "Not known because not 
looked for. But heard, half heard." The white upper- middle-class 
leadership was aware of nonparticipation of Hispanic parents, in this 
project and in the special education services in general. They sent home 
messages in English and in Spanish. They created a plan whereby 
Hispanic parents were telephoned and told about each program, invited to 
attend meetings, and encouraged to ask questions by a person fluent in 
Spanish and English. "We had a Spanish interpreter at a potluck dinner, 
but none came I cannot understand. On the phone they sound so 
interested," recalled an Anglo mother of a SH child. The bottom line of 
all this was, "They just don't come " 

Some project workers, impatient for progress, were critical of the 
call for extra time and energy for recruitment and involvement of Hispanic 
parents. They pointed to honest efforts rtiade to assure that Hispanic 
parents got the information about resources for their handicapped child. 
"Of course the notices should be in Spanish, we aren't trying to keep 
things a secret!" But they felt that "undue" attention to recruiting the 
participation of Hispanic parents "may result in less effort devoted to 
the parents and children ready and willing to use all the help we can get." 

The Mexican-American mother was seen by the frustrated Anglo cs 
having an inappropriately low appreciation of the importance of time. 
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Failure to attend meetings, arriving late, and missing appointments are 

offered as indicators of Mexican-American mothers* devaluation of project 

time. They are seen as not understanding the child's need for special 

resources outside the home. 

I spoke with two Mexican-American scholars at the UCSB Education 

Department about the apparent unwillingness of Mexican-American parents to 

use educational and social services available to their handicapped 

children. They brought to my attention several matters that had not 

occurred to me--and perhaps not to school authorities. They confirmed a 

few perceptions with which I was familiar. They spoke in paraphrase: 

How am I supposed to leave eight kids at home to attend a school 
meeting? 

Look, I am already embarrassed for not knowing the political/social 
system in general. Is it so surprising I would not use the 
educational system, too? 

Maybe, just maybe, the extended Mexican-American family is better 
at supporting SH children than what the school offers, 

A family is supposed to take care of its kids. If a special kid is 
sent to it by God, then it is all the more incumbent on the family to 
take ca. e of it. 

If a Mexican-American parent has to go to the school or other 
agencies for special services it may be seen as a sign that the 
family has failed. Just like the gringos: abandon your kids to 
social agencies! 

If my kid is not normal I have ed. But it is my responsibility 
to take care of her/him. Unless the family has been Anglicized, the 
other children of the family must share the responsibilities for the 
handicapped child. 

If the yringos really wanted to help the SH Me. ican-American child 
they would tear a page out of their manuals for aiding the elderly 

!^r\(\ nivp thp f ;^mi 1 v mnnpv tn hpln thp rhilH hnmp 

The school and its sister agencies could do worse than pay attention 
to these messages in restructuring efforts to get better services for the 
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Mexican-American child in need of special help. The message from these 

scholars, both of whom were born and raired outside the USA, is: 

Schools had better start restructuring programs to meet the 
local conditions. The lack of bilingual staff members to make a 
program run is a pressing problem. Mexican-American parents 
have to be the architects of their own special education 
services network. If parents continue to feel no ownership, 
there is little that transition projects can do to increase 
participation. Communiques, in print or by phone, in English or 
en Espanol , will be met with passive resistance. Federal, 
state, and local programs have to trust local leadership to 
adapt to local condi tions. (Resources are there, if the 
ownership is felt. "You should see the $10,000 weddings that 
come out of the barrio.") 

In every district there is a key casa or two. It is there 
that the meetings have to take place. In safety . It is there 
that the school people must go and take it day by day. Forget 
the long-range planning. What can you do for my child today ? 
Do it. And do something else tomorrow without being asked to do 

it Arrange a second meeting. Talk about what you did for 

that one child that everyone there knows. Then use the VTR 
you brought along to show Mexican-American kids at a camp for 
kids with special needs. . .whatever. Acknowledge that you 
understand their feelings that "God has given you this child as 
a special challenge." If you can't do ihat with honest-to-God 
empathy, don't go to the casa~get someone else who can. After 
really doing something for one of the children in the 
neighborhood, and after showing your VTR, take a family or two 
to see the site of an educational/social service. Don't snow 
them. Just let them look. Talk to them in Spanish about the 
relevance of this for t*^jir child/children. Set up another 
meeting perhaps at another home, if it feels right. 

The two professors had more, much more, about how to approach the 

parents who "just don't come." One thing about which they were in total 

agreement was the need to £a;^ these parents for participating as parent 

trainers, as aides, and helpers. They also recommended that the school 

may want to conduit as much money as it can through the church, the 

Catholic Social Services agency, those who dc have a positive history of 

successful work with recent Mexican-American immigrants. 
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With this soliloquy they ended our interview: "Don't forget, there's 
a good chance that I am illegal, so I have no right to services. Even if 
you tell me I have the right I sure don't want to get sent back. . .again. 
And for sure I am not coming to a meeting in a place that I don't 
understand, to talk about something that embarrasses me, with people I 
don't trust, in a language I can't use. But I will always be polite to 
you on the phone." 

The parent role is a major dimension and a challenge for all who 
design, manage, or participate in transition projects. A second major 
theme that emerged from my interviews is the ultimate in obviousness--so 
obvious as to be missed until I was a month into the study. What is 
transition? 

What Is Transition? 

Each participant in the transition process hes a sense of what she or 
he hopes for, expects, wants. Transition projects appear to offer a 
variety of valuable experiences to a participating student, parent, 
caregiver or administrator. For a long-suffering mother of a SH son it 
means: "Progress! Not much but progress! Before this the sheltered 
workshop was the only non-choice choice. We have been fighting for 
change... and we will be vigilant not to lose what we have won." Each of 
us knows at least some of what we want. 

But the addition of the hopes, expectations, and wants of a second 
person may create a problem, especially if there arc conflicting goals, 
needs, or actions. Problems also occur when participants with 
agreed-upon goals do noc share a vision of the ways to achieve then.. 
There are arguments. 
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Some see t .e way to overcome arguments about defining "transition" is 
to "do it" to a student and point to the successful case as vt. When 
the transition project enthusiast points to a successful placement of a 
20-year-old former student as a case of gainful employment, the listener 
may still remain unconvinced. (If the example cannot be given, the 
doubter is also unconvinced, of course.) An agency administrator dubious 
about the use of the "great example": 

They say they have the employer contacts. It isn't obvious to me. 

You have to be wary of the great example. It may be a tokenism trap. 

"Look what we found on the beach, a jewel! Let's polish it. See 
how pretty it is!" No concern for how many broken shells there 
are on the beach. "Let's polish it!" How likely is it we'll 
find another one? How was it found in the first place? Is it 
indigenous? Is it an artifact of a previous beach party? I 
could go on and on. One great example doth not a (transition) 
program make. 

The "guru of transition," Lou Brown, has a class act. But in 
fact his Stella study had very small Ns. There's a loi: of 
transition B&S without much A&E. [I asked for clarification.] 
"B,A,S,E!" We always seem to have to pretend to have a strong 
base in education before it is there. It's sad. 

For some, transition is essentially getting interagency agreements to 
work. Without them, there cai be no transition. 



Interagency Agreements 

The degree of success attributable to the project's efforts to pursue 
interagency agreements (lAA) depends on the pe. son telling the story. The 
project's directors featured lAA in my in-task interview, as did the 
chairwoman of the project's steering committee. Roughly the ascription of 
success parallels the amount of involvement with lAA on the part of the 
informant, with a few notable exceptions. The project leadership 
indicated that 

the cooperation we have achieved will be permanent. What used 
to be a ritualistic sign-them-and-forget-them attitude toward 
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interagency agreements has been changed due to this project's 
efforts. They are more than formal paper shuffling. There's 
life in those documents now. That is a structural change that 
will outlive this project. 

This view was echoed by the Chairwoman of the Steering Committee, She 

said that a business and school alliance had been started that was 

important and would be durable. 

We have formed an alliance here that can serve as a model for 
interagency agreements. We have taken a marketing approach that 
the school people never have taken. That assures us of 
success. If you can't sell your product to the other agencies 
they aren't going to buy it. It's just that simple,, ,1 think we 
have been very successful in our interagency agreement 
effoit. There is more to be accomplished, I am confident we 
can do it. 

Others in the school district were less certain of the extent of 
interagency cooperation. They had not been involved directly in the 
district's efforts to create meaningful lAA's, They said they felt good 
work had been undertaken and that t.ie results had been positive. However, 
several of these "hearsay" witnesses went on to comment on the "poor" or 
"spasmodic" or "fading" participation of industry/business 
representatives in the project: "I never see the same agency people 
twice, " 

They started going, then at a later meeting someone replaced 
them as the representative, and finally no one came, I don't 
think it was anyone's fault. They are just too busy to 
participate in meetings. 

The view of the lAA component of the project from non-school 

personnel was more critical. One agency representative harshly observed. 

The lAA effort? Oh that's easy to summarize: wrong from the 
outset, ,, doomed to failure. The contributing factors were 
several. First, the wrong conceptual moae i ; second, the wrong 
style of presentation; and third, the wrong cast of characters. 

This person works in a non-school agency serving populations with 

special needs. Offered in an objective manner, her observations built 
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upon the central ity of her "doomed to failure" feeling. "The lAA failure 
could not have been otherwise, given its school oriented leadership." She 
introduced me to another person who had attended a project meeting or two. 
He was "turned off by being told to 'Put on my thinking cap.' What the 
hell is this: an education/industry operation or ding-dong nursery 
school?" Another corroborated being put off by "teacherly" behavior, 
interestingly enough not on the part of educators but by a key business 
representative serving on the Steering Committee. 

My gut feeling is that the business/industry component of the Santa 
Barbara lAA was a moderate success. I wish I could have gotten someone to 
open up on interagency problems, antagonisms such as vocational 
rehabilitation getting money for closing cases of kids who still need 
service. A key question might be, "Who owns these kids?" For four weeks 
I tried with no success to schedule two interviews with business 
representatives nominated by the project. Finally I heard from one that 
he "really hadn't attended enough meetings to know very much about the 
project." And the second called on the very day to cancel a long-standing 
appointment, observing that "others would be more helpful and informed 
about the Transition Project" than he. 

The comment about lAA conceptual model "error" was provocative, 
leading me to interview an informfnt about another California transition 
project. The assertion was that the SB project's lAA failed because it 
began, "ran on and on, and died at the ideational level," whereas the 
other project (reputedly) began its lAA work with a handful of senior 
students and worked exclusively on answering questions about who had to be 
involved to move these students from school into the workforce. The lAA 
effort may have buffered from non-participation of major non-school 
agency decision-makers. 
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The industry non-participation Issue aside, clear s^rides forward 
were claimed by many for the lAA efforts. Early doubters were converted: 
"I thought: the lAA would get lost in the Tri-county bureaucracy for sure. 
Not so. In fact, other high schools are modeling programs after 
S.B.H.S." Another participant attributed lAA success to the 
"participation of live parents attending meetings. The mud slinging 
lessens. Finally 10 years after the law we get cooperation instead of 
competition! Credit the parents for some of that... Now the agencies at 
least have to appear to be working with one another." 

The evaluation of a transition project should meet the question, 

"What is transition?" head on. It will be raised separately and 

disharmoniously by key players in a local setting. Despite the difficulty 

of developing a harmoniously shared view of transition, it appears 

possible to start and work hard on a transition project before forging a 

single definition. Some feel it may be preferable to begin a transition 

project without attempting to hammer out an answer to the question, "What 

is transition?" An evaluation planning specialist in the California state 

education agency had given this much thought. 

From my perspective it is impossible to move on this (statewide 
transition effort) until there is an agreement on the definition 
of transition. An operational definition that links resources 
to programs to outcomes. But we can't assess the quality of X 
until we know what X is. It's funny s though, in some odd sense 
you can get going on X...on the spirit of X without knowing 
exactly what it is. . .progress has been made in several districts 
with very loosely constructed statements defining transition.... 

But it is not useful to evaluate these efforts until we (the 
state) and they (project directors) get our separate acts 
together. Transition is supposed to kick down the door between 
school and life. But who is responsible for what, when? l/ho 
has follow-up responsioil ity? Which data are to drive which 
decisions? We had 21 OSER projects in *85, nine in '86. We'd 
like to disseminate the 'best" transition project results 

Transition is a general term for a lot of things. The state 
agency is couing to focus on outcome expectations such as 
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employmeri:, 1 i ving conditions, interpersonal relations and 
integration into the communi ty. These orient all our evaluations. 

But it is too early to tell. The whole effort is in 
"transition." We have a present objective of getting a shared 
vision. 

His quintessential goal of a "shared vision" was echoed by several 
state regional and local special educai.on administrators with transition 
responsibilities. No one saw a role for evaluation in this develop- 
mental aspect of transition projects. When I suggested the possibility of 
having an evaluation consultant assist them in clarifying state, regional 
and local answers to the question "What is transition?" I received reac- 
tions ranging from incredulity to puzzlement. There is an openness to 
hiring consultant help but evaluators need not apply. This is the domain 
of the "process facilitator," the O.D. specialist," or the "educational 
therapist"--not the evaluation specialist. "We need help getting compe- 
ting agencies to put away the boxing gloves more than we need program 
outcome evaluation" said a SERN specialist. "Evalu.^tion is by nature 
hurtful." Here, as elsewhere, I found very few who saw educational 
evaluation as a helpful experience. A few were open to the possibility of 
such occurring but thought it unliKely. "Evaluation should tell us 
what's good, what's worth sharing, and sometimes how to repair something 
worth saving," observed a state agency planne.-. "What it usually 
accomplishes is more like research; it ties dollars to programs to 
outcomes, if it is really good (eval uation) .. .usual ly there are no 
outcomes. "*" 



ERIC 



2. Could the Tl/UI produce a set of 'What is transition?" probes with a 
companion set of programmatic consequences for local, regional, 
perhaps state use? Could it be in a Fred Goodman (U. of Michigan) 
game format? Could it be grist for parent-training mill workers 
and/or for groups of mixed role players struggling for a shared 
vision of what transition is? 

80 
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Parents, the idea of transition, and evaluation are the three large 
topics of this report. We turn now to the third, evaluation. 

What About Evaluation? 

For the most part evaluation has not been seen as a vital force in 
transition projects. Because t^ere is almost no two-way communication 
between local projects and the federal agency, the promise of evaluation 
(for local participants) is marginal at best. It will be an "add on," 
unrelated to the life, development, or restructuring of a project. As 
attempts are made to make project evaluations formative or responsive to 
federal initiatives, they will be burdened by rapid turnover of project 
o'^ficers in the federal agency. 

The best the federal agency can hope for is a sensible compliance 
checklist that addresses key accountability issues, one that doesn't 
attempt to captu'^e the value dimensions, the decision aspects, or the 
contextual relationships of local projects. Those things take expertise, 
energy, and local commitment, investments that cannot be matched at the 
federal level. 

How People Check Transition Project Evaluations 
During an early interview, a former resource teacher talked about her 
experiences with educational evaluation over the years (22 of them). As I 
reached for my interview questions in my briefcase, she pointed to a 
largish volume I had in my briefcase and said, "I thought you said a ftw 
questions!*' I nl;^yeH ^long and facctiCuSly Ciaimed, "Tliib happens to be a 
certified evaluation of the most successful transition project in the 
United States. Would you like to see it?" 
"You're kidding?" 
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"If I am not, what would you be most interested in seeing in the 
report?" 

"The first thing is: Who was involved? Were parents there? Were 
key agencies?" 

The unplanned tactic became one of the more productive l^nes of 
Inquiry throughout my interviews. The following are several of the 
responses, followed in parentheses by the interviewee's role in the 
project. 

The first thing is, who was involved? Were the key agencies 
really a part of the transition? Parents want to know that 
their Kids will not drift into institutions, the streets [after 
high school]. Does the transition project have a structural 
framework that per. . .encourages kids to come back after they 
have left school? Is there a limit to the number of times? 
That's what I'd look to see. If those aren't there, I wouldn t 
pay too much attention to the report. (County Administrator) 

I'd like to see the curriculum. Does everything relate to 
employability? If it does, I won't think much of it. I know 
this is heresy, but everything in the curriculum doesn t have to 
[connect with employability]. How about a student's 
self-esteem? "Hey, I can learn things 1" That s important 
too A curriculum that helps you'>gsters understand connections, 
relations... my (the learner's) power... what a community is. 
That's a curriculum for transition. Has there been a balance 
between filling out job forms and knowing about knowing? 
(Resource Teacher) 

Each subsequent interviewee was given an opportunity to express what 
she or he would look for (or does look for) in an evaluation report. Each 
person found the question easy to answer. 1 tried to preserve the order 
in which they itemized what they would do with such a report and listed 
their comments in 1, 2, 3, 4 order, even if they did not so enumerate. 
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Before turning to the verbatim responses, one overall observation 
about this group's familiarity and experienci with "a good evaluation 
report" is warranted. Twenty- two of 30 had neve^^ seen what they 
considered a "good" evaluation report. Four said they had never seen an 
evaluation report. Of the remaining four, two had seen a "good" one, 
(One of them had written the only good evaluation she had seen.) Even sc 
their collective responses reveal much about the perception of evaluation. 

An administrator in a neighboring district said he had seen "one good 
evaluation effort in 31 years... It was a highly behavioristic project with 
simple data collection. Low inference. No extra claims." 

Seems to me we have a lot of work to do. Here now are the 
encapusulated replies to my indirect question of what to look for in an 
evaluation report: 

1. The Introduction: What it is setting out to do. 

2:. The Ending: How it all naopened and if in fact it did happen. 

3. Then I'd look to see if they measured the right things. 

a. Outcome tables 

b. Did they evaluate orocess? 

c. Did they measure the feelings of the participants? 

4. I'd check again on the process measures-the committees, the strategy 
meetings, the networking. 

5. Do they have a representation of the quality of cooperation? If the 
cooperation isn't there, long-term continuity won't occur. 

6. Finally, the bottom line: how did ihey get the students from 
unemployment to employment? The kindr> of placement, duration. "We 
wouldn't look good on that here." 

7. Evaluations are typically for the funding source. They don't 
represent the reality of ihc day-to-day going- on and outcomes of the 
local project. If I know this Is a report for the feds to show you 
have spent the money wisely, I am not interested. It doe^sn't 
translate for me. 

***** 
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1. What do the parents say? Especially the reputable parents. 

2. What do the students say? Is there evidenc" that it worked (in their 
eyes)? 

3. rd see who was involved. Was it really jusc the district? Is it 
another project in the school's own little world? 

4. Then I'd look for their plan for transition. (No, I wouldn't turn to 
that first.) 

5. '^oes the report f'cknowledge that this is the tip of the iceberg? 
This is what helps the most right now. ..this will help in the ^Qr.^ 
run? 

1. Of course I'd look for the only thing that really counts: the 
effectiveness of placements. How were the students prepared for 
them? Were they tracked and is there evidence of success? 

2. Look for the quality of life--not just goal statements. Are they 
basically pursuing a career in the world of work? And do they stay 
there for many years? 

3. Student numbers impress me. Anybody can cream a few and look good. 

4. rd look for lack of duplication of services between the agencies. 

5. Look for indKators of working cooperatively--not competitively 
between the < ^^ncies. 

6. For sure Td check on evidence of parental awareness of the need for 
transition services, accessibility of help, on their participation in 
the lEP development (ITP). 

7. And I'd look again at the creaming. It's one thing to take resource 
specialists' kids and place them on jobs versus profoundly 
handicapped kids. It's hard to get "good numbers" when working with 
the full range of kids. 

★ ★ ★ ★ ★ 

1. Perhaps evaluation could take an Edward DeBono approach. In all 
likelihood I honestly wouldn't read an evaluation report. 

2. If I did, I would sweep broadly and quickly for something of interest. 

3. Sort it out into pluses and minuses for the players in the game: the 
teacher's, parent's, child's. 
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4. That's what ^'d really look fo'-: what is the quality of life for 
these kids? Is there real people evidence that shows the 
effectiveness of the services for the recipients? 

!). When is educational evaluation going to discover visuals, graphics, 
videotape? The narrative should be supported by visual displays of 
the life that has supposedly improved, 

* Vr /c * ★ 

1. Why did they want to do it? 

2. What did they do? 

3. I would avoid the statistics completely. 

4. Look for the unmet needs, what was not being addre3sed, what they 
avoided. 

5. I like to look for the difference betwean the flowering i.eginning and 
the botLoni line summary's factr . 

***** 

1. Quickly scan the table of contents. 

2. Speea-read the preface and the introduction. 

3. Get to the results as seen by the operators of the project--not the 
evaluation experts. 

4. I'd read it the W3y I read a journal: scan, focus cn the abstract, 
Ihe results, and then dwell on anything that catches my interest. 

***** 

1. I have worked with schools on this very topic and have a few strong 
opinions. First: the simpler the better; for example, does the line 
go up, or does the line go down? Second: the school's instinct is 
not to evaluate. They are interested in the absence of complaints 
about a project. That is the criterion of success, not outcome 
measures. That is followed by teacher improvement, excitement by the 
project. When the public is not bitching and the teachers are cooing 
you've got a successful project. Oh, add to that the ego 
gratification of the director. Be sure not to mention "research" or 
"evaluation'^ when you consult witn c transition project. When they 
hear "research," do they run! 

2. I would first Xerox a copy because you said it was "exemplary," and I 
would finally have one in my library! 

3. I like fancy analyses. I like research with multidimensional scaling 
that pictorial ly represents concepts. 
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I'd look first at the table of contents to get the sophistication of 
the design, I'd examine one spot to see if It was merely a pre-post 
test design. 

5. I'd look to their conclusions to see their claim? and how they got 
the information to support the claims. 

6. A quick check of what the authors claim are important outcomes is 
always a good test to see if you want to read more. 

it it it it it 

1. I always check to see who was irivolved. If I don't care who was 
involved, I dcn't care about the report. 

2. Did they find out the parents' hopes and beliefs about the outcomes 
of the project? Can the kids come back? Is there a limit to the 
nun;ber of times? 

3. WhaL is their concept of curriculum? Everything in the TI curriculum 
doesn't have to relate to employability. How about the students' 
self-esteem: "Hey, I can learn!" Does the description of the 
curriculum convince me that it helps students to understand 
connections? Relations? My power? What a community is? All in all 

I want to see a curriculum that balances bevween filling out forms 
and keeping a job and knowing about knowing. If that isn't there, I 
don't care anything about the evaluation report. 

★ ★ ★ ★ ★ 

1. I want to read the part that talks about success two ye^»^s after 
being on the job. One hundred percent placement at the Lid of the 
program is fine--but it's not my criterion of success. 

2. I tend to be cynical about any form of statistics. 

3. If a report could be done that allows me to meet a family... the 

families of clients themselves The videotape portrayal of the 

family could help me immensely to understand, to evaluate the success 
of a program. 

***** 

1. Student outcomes, which is really "job placements." That*s all that 
counts. Longevity on the job and the quality demands of the job. 

2. Description o^ living arrangements for kids. 

3. The client's use of leisure time is something I'd look for. 

4. Maybe I'd look for indications that the transition players' efforts 
to serve mutual clients had eliminated antagonistic intakr processes. 

***** 



ERIC 



£6 



Model Programs 78 



L You've got to check on the number of students processed, placed, 
evaluated: the end proauct. 

2. All of the meetings wouldn't mean a thing to me, I want to know now 
the cake tastes. The ingredients don't count, 

3. Follow-up data four years after placement would be nice, 

* * * ★ * 

1, Which jobs were involved? 

2, Are the placements over-ambitious? 

3, jj/ho did the follow-up insofar as coaching and support was concerned? 

4, What do the real parents say? (Parents of SH students,) 

***** 

1. You've got to first figure out their terms. What counts as informa- 
tion? 

2. Who collected :he information? 

3, How did the project get its results? Its techniques? 

4, Then I'd stop and ask myself, is that something r_d want to achieve? 

***** 

1, Look at the kids that were placed What were the limits? The 
successful project takes the full r.rnge of kids, 

2, Look at teachers' expectations prior to the project. 

3, Were the parents' expectations too high? Too low? 

4, Evidence of administration overly controlling the project? Did the 
parents push it? The teachers follow? 

5, At least a hint that the success the project is going to take credit 
for is stuff that hcs been going on for years, 

6, How does the T«project actually run? How did they get by the 
obstacles? Prejudice against special kids? Industry's unwillingness 
to take a chance? How to get l^ng-term support? 

7, Discussion of failures, [If] no failures, don't believe the report, 

***** 
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First thing I'd look at are the characteristics of the population 
served. How severe are the disabilities? One can look good by 
creaming. 

Second: the number served. If it is in L.A. and they served six, 
they could look good but undeservedly. 

The kinds of employment options made available. Merely custodial? 

The mean dcllars income of the program's clients. If it is a job 
development transition project, is it more than token work? 

How does (or doesn't) it address the non-work of people's lives? 
People don't totally work to live. Does it address the life style of 
clients during non-work hours? Contribute to the stability of their 
residential lives? Their physical well being? 

Do students know how to function in the community? Can they get to 
work on time? Can they care for their bodies? 

I pay very little attention to program descriptions. I have not seen 
a single non-self-serving evaluation of special education programs. 
They are gloss. Ninetv-percent b.s. 

I look for forms, ideas. How did they bring parents into the 
picture? How did they deal with the Department of Labor? How did 
they wedge the student into a reluctant industry? 

Program descriptions are written for funding, refunding, donations, 
referral. There just isn't critical stuff available. What is 
available is East uoast. . .Virginia Conmonwealth. ..a bit from '!Y (WA 
or OR). Moss, Bel lamy . . .but theirs is a sales job, too. 

Its credibility is high when I see the author address a problem that 
I recognize as a problem. 

***** 
I'd zero in on the narrative outcomes first. 

I turn to statistics only when I can't figure it out from narration. 

Sometimes I look at what they were trying to accomplish. 

Were the critical questions raised and were Lhe project's answers 
approached in a reasonable manner? 

***** 

Well, you have to forget the evaluation itself. You can get suckered 
into the Form of it. I couldn't care less about whether they u.ed an 
experimental design, were systematic this or that. Is there 
anything I can use? 
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2. Does it reveal a problem I care about? 

3. Does it suggest overall and local approaches to remedying the problem? 

4. Percentages will do just fine. Don't bust your ass deepening and 
mystifying. 

5. I don't care whu did the report. Are they available for interpre- 
tations? 

6. The least valuable section of these educational evaluation reports 
are the 700 tests of the significance of the findings, and absence of 
straightforward, doable commentary. They kill my inte est. 

7. There is use for literature narrative. The New Yorker carried an 
anecdotal report on social service abuses. Nothing that lent itself 
directly to significant social policy development--but it alerted us 
to pitfalls we could create in our legislative development. . .things 
that would frustrate implementation. It helped as an antidote to 
pol icy-promise-delusions. 

8. Evaluation reports can be useful if they help identify people who 
might be smart enough to help me in my tasks. So I suspect they 
should carry something about the creators' skills, interests, 
availabil ity. 

***** 

1. I can't even imagine opening the thing. 

2. OK, I'll imagine. Does it say anything about how to get people to 
use other people's work in education? 

3. Does it have something besides testimonials without outcome data? 

***** 

1. I always start at the end: read the conclusions and interpretations. 

2. Then I go to the beginning and see if the intentions are congruent. 

3. Quickly sense the methodological mechanics, quick sense of the 
numbers. 

4. What's the connection of the goals, to the change strategies to the 
effect; on the clients? 

5. I would not read the executive summary. (I was shocked! TD) They 
are too clipped. They have to be politically palatable. 

6. If there were a case study included, which is unlikely, I would go to 
its conclusion as well to look for leaps from the case to the larger 
issue(s) which I would hope were there. If they are there, then I'd 
look at the case study. 
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7. Something that isn't in this report but which I keep waiting for is a 
Beta Max documentary with accompanying hard copy. 

★ ★ ★ ★ ★ 

1. I \ouldn't look at it. I'd have someone else do it for me. I'm 
tired of measuring. 

2. Well, I'd tell the person to see if it was something we can use. 
Does it match with our current questions. W2 don't need more damn 
reports about SH kids not being employed. We don't need wonderful 
descriptions about classroom life for the SH kid. 

3. Where are the kids? And are they out doing it? If you tell me 
that, then and only then will I want to know more. 

4. If an evaluation revealed the mechanics of enhancing the process of 
achieving a shared vision of what transition is I could get 
interested. 

★ ★ * ★ ★ 

1. The length of the viuv is stop number one. If the project hasn't 
got several years I'd be less than impressed with any of its findings. 

2. Who have they claimed to serve? How many? How placed? 

3. What I am always hoping to find is an R.B. Edgerton (UCLA sociolo- 
gist) description of what really happened. Have you read the Cloak 
of Competence ? When the California institutions were closed by 
Governor Reagan, Edgerton did a fcllow-up study that revealed the 
culture, the lives, the magazines they were reading. . .the social 

events It was so real... I am so visual. I am kinesthetic It 

was Faulkner. I had a feeling for the quality of that former 
resident's life. I remember the description of the guy's ordering 
copies of Life and Readers' Digest to neatly array on his table. 
He couldn't read but he sure knew how to take on one task in this 
world; how to "pass" as a reader. . .achieve normalization. 

Sometimes you experience a painTul level of candor when an informant 

decides to level with you. Here's oner 



You want to know the real problem with evaluation? You are! 
The problems of transition are far away from the grant writers 
and the proposal readers and the visiting expert evaluators. 
You can't attack them at that level. 

Everytim^ a new grant occurs or an expert visits expectations 
are raised. When a pf^rson needs something, anything positive 
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you say or do raises expectations. Once again you start to 
believe, "Gee, maybe we are going to get some help," But real 
problems are enduring. Visiting experts and two-year grants 
don't help. It's people in the trenches day in, yecr out, that 
make the sobering difference. Some of us need hope. Some need 
a sobering thought The balance between them is important. 
That kind of stuff can never be in the minds of grant writers 
and visiting experts. I mean no harm to you personally. You 
just represent something I happen to think doesn't help. 

An old pro in the state agency summed up several impressions of what 

others had said about the role of program evaluation in special education. 

Projects are afraid to cooperate witn evaluators. They are 
dfraid of losing funding. They are afraid they have made 
errors. They are afraid they have not done the job. So we get 
a lot of hesitation, even refusal [to cooperate ;n 
evaluations]. We (SDE) need to stop tying dollars to failure, 
tying money to avoiding making a mistake. If you chop off 
funding without finriinn how the mistake came to be you miss an 
opportunity (for understanding program development) 

"We encourage self-deceit, over-claiming, smoke screening. It 
shows up even in the well established 'corner scone' centers of 
the movement. It makes me wonder if their images are well 
deserved. " 

What is to be made of all this? Much, I hope! Is there a pattern 
discernible that can be put to use in evaluation improvement? Is the 
surprising (to the writer) call for narrative and qualitative 
representation by the participants in this transition project a 
consequence of the writer's bias or something independent of that? I 
recommend other fieldworkers ask this question in their work and see if 
their results parallel mine. 

One anomoly is apparent, for example. People in this study appeared 
to be interested in the topic of context being included in evaluation 
reporting. (Only one person did not appear to think it was of 
considerable importance.) Out of these discussions surfaced ideas for 
possible use in capturing the values in local context (e.g.. Chamber of 
Commerce, voter analyses, novelists, portrayals). 
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But when we played the garble of quick review of an evaluation report 
devoted to an excellent transition project, not one interviewee turned to 
context first, and only one turned to it second or third. 

What can be made of this? Three explanations at least can be 
offered. It could be that the match between exotic and local context 
isn't very important for them as evaluation r^eport readers. Or, the 
opportunity for assessing such a match is so rarely provided by evaluation 
report writers that it doesn't come readily to mind. Or, the "good" 
evaluation report simulation wasn't a fair test of the proposition. 

Desides the three major concerns of this story (parents, transition, 
and evaluation), we have concerned ourselves with context. As is always 
the case with stories of educational practice, minor themes come to view, 
themes we are unable to wedge nicely into the story. Often we are unable 
to discard them. What follows now is a coterie of such themes. 

Melange 

Coming to know new people (or oneself anew) was acclaimed by several 

as a positive side effect of participating in the project. Fo.- some, the 

many meetings served as social connector as much as facilitator of 

transition project activity. Indeed, for a handful of my informants, 

connecting with new persons was the -^avorable outcome of their project 

involvement. This was particularly evident when encountering a special 

attendee having high status, professionally, socially, or politically. 

Seeing him attend really impressed me. He is seen as the czar 
of psychological services for the tri-county area. His com- 
mitment caused me to keep attending. I would have dropped out. 
He is a top quality player. Getting to kn^-^w him better made the 
meetings, which I must honestly say were disappointing, ...If 
they had been able to attract and keep other top quality players 
involved they would have come close to really doing something 
with interagency agreements. 
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Thus spoke an administrator from one of the nonschool social service 

agencies. Another informant, a teacher, praised the project for providing 

much needed social opportuni ties. 

We now know one another better. Before the project I knew some 
of them by name only or had maybe heard about them. Now I know 
what they like for lunch, that their youngest son has Downs, how 
they feel about sheltered workshops, I think the project served 
a purpose beyond its original purpose. It wasn't supposed to 
especially enable the players to get to know and trust one 
another. But that happened here and there. And I feel that was 
the most important thing it accomplished for some of us,,, of 
course some people didn't attend meetings so we didn't get very 
far with them or their agencies. 

For one the project was an opportunity to recharge an inert 

special -education- teacher career, 

I was stifled, unhappy, ,, felt useless. I'd like to think of 
myself as a potentially creative person outside the classroom, 
I hoped I would get an idea about doing something from this 
project, 1 was ready to break out of the doldrums. Well, it 
happened! This project let me cut acj oss many areas, I got 
into job coaching, computers, curriculum work..,! became chair 
of the Language Arts committee which included math. We have met 
3-5 hours monthly and there have been lots of individual hours, 
I have been an active professional before earlier in my career 
(recounted activities of 15 years ago),,. My career is now 
revived. People see what I can do [because of her project 
work]. It sort of evolved. The more I did, the more I saw how 
it was working out. 

Arid finally we have the steering committee chair, a high school 

graduate who has worked her way up the ladder in a statewide banking 

system to Director of Personnel. She is in her mid 30s, is obviously 

pleased with her accomplishments and thoroughly enjo>s the interpersonal 

dimension of the project. 

The run part of the project is the chance this has given me to 
talk, I can't stand silence, I like to finish things, I like 
to get school people to see how to market ideas. It's the same 
as selling merchants' Master Charges, Find out what keeps them 
busiest? What would save them time? Get going, give them 
service if it bre:.k^ down,-, and find out what else we can do for 
you. I like teaching school people about business practices ,,, I 
guess I could say I enjoyed that [teaching] the most about the 
project. 
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Others echoed the "connections" theme in their recounting the benefits 

of the project as they saw them. For most, the emphasis given to 

self-renewal or interpersonal relationships was not as great as it was for 

those represented above. But the general importance cannot be denied. 

Projects serve to advance personal and interpersonal agendas as surely as 

they do achieve their espoused goals. On occasion the personal intents 

appear to be the story of an individual's participation in a project such 

as this. The life of one junior high teacher was greatly enhanced: 

We enthusiastically worked on the curriculum even though the 
project was focused on the high school. So much free time was 
given. I've had so many inservices before that were no fun. 
This was. Great exchange of ideas. We aren't institutionalized 
yet [as to project outcomes] and we need more experience working 
with these kids. But the barriers are ignorance, not mean 
spiritedness, and economics (paid job coaches), not lack of 
vision We now need a lot of job niches, not committee meet- 
ings. I think we're on the edge of something very important 
here. 
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